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PREFACE 


In  1970  and  1971,  Rand  conducted  a  critical  evaluation  of  the  ac¬ 
tivity  and  products  of  gaming,  model-building,  and  simulation,  under 
the  sponsorship  of  the  Defense  Advanced  Research  Projects  Agency.  The 
specific  aim  of  that  inquiry  was  to  assess  the  usefulness  of  gaming  in 
military-political  policymaking.  Its  general  aim  was  to  contribute  to 
the  definition  of  common  standards  and  the  refinement  of  objectives 
that  are  necessary  to  the  advancement  of  the  gaming  profession. 

As  part  of  that  study,  the  authors,  in  cooperation  with  the  U.S. 
General  Accounting  Office,  developed  and  administered  a  detailed  survey 
instrument  to  DOD  personnel  knowledgeable  about  a  considerable  subset 
of  models,  simulations,  and  games  currently  used  in  the  DOD,  This  re¬ 
port  describes  the  survey  procedure,  presents  the  results,  and  discusses 
their  significance.  (The  GAO  is  independently  analyzing  the  results.) 

The  discussion  covers  the  purposes,  production,  operation,  use,  and 
costs  of  the  132  models,  simulations,  and  games  surveyed.  Respondents 
opinions  about  several  hypothetical  innovations  in  the  gaming  profes¬ 
sion  are  reported.  The  survey  instrument  is  appended. 

Based  on  their  analysis  of  the  survey  results,  the  authors  advance 
their  own  recommendations  concerning  the  following  aspects  of  the  pro¬ 
duction  and  use  of  models,  simulations,  and  games:  advocacy  versus 
scientific  validation;  costs;  professional  reviewing  and  standards; 
documentation;  redundancy  and  standardization;  clearinghouses;  research 
needs;  sizes  of  models,  simulations,  and  games;  free-form  and  man-machine 
gaming;  and  gaming  in  the  civilian  sector.  Taken  together,  the  recom¬ 
mendations  indicate  several  ways  in  which  increasing  professionalism 
can  benefit  both  gamers  and  the  policymakers  they  serve. 

Other  Rand  publications  deriving  from  this  research  include 
R- 6 20- ARP  A,  The  Literature  of  Gaming ,  Simulation,  and  Model-Building: 
Index  and  Critical  Abstracts,  by  M.  Shubik,  G.  Brewer,  and  E.  Savage, 
1972,  and  R-732-ARPA,  Reviews  of  Selected  Books  and  Articles  on  Garmng 
and  Simulation,  by  M.  Shubik  and  G.  Brewer,  1972. 
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I .  INTRODUCTION 


These  little  soldiers  marching  out 
Could  put  the  bravest  foe  to  rout 
And  place  the  world  beneath  their  sway; 

But  since  their  arms  are  rather  light. 

If  I  interpret  it  aright,  * 

They're  only  setting  out  to  play. 

CLASSES  OF  MODELS,  SIMULATIONS,  AND  GAMES 

Complex  problems  often  demand  complex  analytic  techniques:  gaming 
and  simulation  are  two  such  methods.  Generally  characterized  as  having 
not  only  intuitive  appeal — based  on  nice  descriptive  properties  but 
also  a  solid  empirical  basis,  these  methods  have  stimulated  the  devel¬ 
opment  of  a  large  and  expanding  professional  community  and  literature. 
What  this  community  does,  the  trends  and  size  of  its  activity,  and  where 
it  seems  to  be  headed  are  questions  of  importance. 

Even  after  many  years  of  work,  it  is  still  difficult  to  state  with 
precision  what  is  meant  by  gaming  and  simulation.  What  start  out  as 
games,  for  example,  may  degenerate  into  fruitless  syntactic  exercises. 
However,  it  is  useful  to  distinguish  four  categories:  analytic  models, 
machine  simulations,  man— machine  games,  and  free— form  games. 


Analytic  Models 

A  good  analytic  model  is  usually  quite  abstract,  poor  in  the  num¬ 
ber  of  variables  explicitly  considered  but  rich  in  ease  of  manipulation 
and  clarity  of  insight.  For  many  questions,  the  analytic  model  may 
give  a  single  number  for  an  answer,  as  contrasted  with  multiple,  inter¬ 
related  indicators  of  system  behavior  that  may  result  from  the  use  of 
other  techniques.  True,  more  than  a  single  number,  a  kill  probability 
or  a  specific  survival  level,  for  instance,  may  be  of  analytic  interest 
Frequently,  however,  a  single  end-state  condition  for  a  system  is 

* Stanley  Appelbaum,  trans.,  "War,"  Games  and  Pastimes  of  Child¬ 
hood ,  New  York:  Dover,  1969,  pi.  26;  originally  published  as  Jacques 
Stella,  Les  Jeux  et  Plaisirs  de  I'Enfanae ,  Paris,  1657. 


calculated.  This  contrasts  with  the  study  of  system  behavior  through 
time,  in  which  end  states  may  not  be  of  interest  or  even  relevant. 

Certain  forms  of  warfare  have  been  characterized  by  game-theoretic, 

analytic  models  in  which  two-sided  or  more  than  two-sided  combat  is 

* 

considered  explicitly.  For  example,  one  might  wish  to  examine  the  op¬ 
timal  behavior  of  a  red  team  versus  a  blue  team  when  neither  side's 
strategy  is  fixed.  In  a  situation  such  as  a  two-sided  war,  one  would 
think  that  assuming  pure  opposition  would  enable  one  to  consider  opti¬ 
mal  tactics,  and  that  deductions  of  an  opposition's  best  strategy  could 
lead  to  the  formulation  of  optimal  strategies.  Unfortunately,  it  is 
generally  not  possible  to  do  so.  Combat  situations  more  complex  than 
simple  tactical  encounters  are  frequently  not  well  represented  as  zero- 
sum  games"  because  they  may  not  be  situations  of  pure  opposition.  Fur¬ 
thermore,  elements  omitted  from  the  analysis  in  the  interest  of  tract- 
ability  and  precision  (for  example,  "human  factors")  may  be  crucial  to 
understanding  what  is  in  fact  going  on. 

Good  analytic  models  help  spot  the  "chicken  and  egg"  problem  that, 
once  recognized,  can  usually  be  solved.  The  point  bears  directly  on 
the  relationship  among  rigorous  theoretical  models,  empiricism,  and 
data  gathering.  An  analytic  model  is  usually  too  restricted  to  solve 
an  actual  operational  problem  directly.  But  because  a  model  is  norm¬ 
ally  clean  and. clear,  it  can  warn  about  potential  difficulties,  indicate 
where  additional  measurements  are  most  needed,  and  identify  and  order 
important  omissions.  This  presupposes,  of  course,  that  those  preparing 
the  analytic  models  communicate  with  those  who  have  the  operational 
problem,  and  that  their  findings  are  received  and  understood. 

The  optimal-assignment  method  created  by  John  von  Neumann  is  an 
excellent  example  of  an  analytic  model  that  has  limited  interest  for 

For  example,  Melvin  Dresher,  Games  of  Strategy:  Theovy  and  Ap¬ 
plications,  Englewood  Cliffs,  N.  J.:  Prentice-Hall,  1961. 

^ John  von  Neumann,  "A  Certain  Two-Person,  Zero-sum  Game  Equivalent 
to  the  Optimal  Assignment  Problem,"  in  H.  W.  Kuhn  and  A.  W.  Tucker, 
eds.,  Annals  of  Mathematical  Study ,  No.  28,  Princeton  University  Press, 
1953,  pp.  12-15. 
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direct  operational  application  but  that  provides  many  basic  insights. 
Assuming  that  one  knows  exactly  what  a  man  can  and  will  do,  that  model 
considers  a  set  of  tasks  that  must  be  done  and  offers  a  way  to  assign 
men  to  tasks  in  an  optimal  manner.  Though  the  simplifying  assumptions 
may  reduce  the  model’s  direct  operational  application  and  utility, 
von  Neumann’s  work  is  clean  and  abstractly  captures  the  core  of  an  im¬ 
portant  problem. 

Analytic  work  on  the  Berlin  airlift  provides  an  example  where  ini¬ 
tially  simple  analysis  yielded  to  more  complex  formulations  as  the  prob¬ 
lem  began  to  be  understood.  From  the  first  linear  program,  the  anal¬ 
ysis  evolved  to  dynamic  models  and  ended  up  as  a  combination  dynamic 
model  and  simulation.  The  evolutionary  process  was  probably,  in  a 
strict  pragmatic  sense,  optimal.  One  might  conjecture  that  until  the 
problem  was  ’’learned”  with  simpler,  more  abstract  analyses,  realistic 
representation  was  just  not  possible. 

People  most  at  home  with  analytic  models  include  applied  mathema¬ 
ticians,  operations  research  analysts,  and  a  breed  that  may  be  called 
’’computerniks,”  although  the  computer  is  frequently  used  merely  as  an 
analytic  aid. 

Growth  in  the  use  and  power  of  analytic  models  over  the  last  30 
years  has  been  astounding.  In  techniques,  new  insights,  and  amount 
of  personnel,  that  growth  has  probably  exceeded  all  previous  work.  The 
prospects  are  good  for  more  diverse  applications  of  analytic  models  and 
for  increased  use  of  the  computer,  not  as  a  simulator  but  as  an  aid  in 
solving  analytic  problems.  The  outlook  is  bright  for  serious  model¬ 
building,  problem-solving,  and  careful  application.  The  status  of  and 
prospects  for  the  three  other  categories  are  less  certain. 

Machine  Simulations 

In  contrast  with  analytic  models,  machine  simulations  frequently 
involve  many  variables;  many  seem  to  make  a  fetish  of  ’’realism.”  Ra¬ 
tionales  for  doing  simulations  are  many  and  varied.  One  common  and 

*A.  S.  Manne,  Allocating  MATS  Equipment  with  the  Aid  of  Linear 
Programming ,  The  Rand  Corporation,  RM-1612,  January  1956. 
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frequently  valid  reason  is  that  mathematics  is  a  relatively  impoverished 
language,  whereas  the  computer  allows  one  to  capture  the  richness  or 
robustness  of  a  real  system.  What  is  left  unstated  is  the  enormous 
price  one  is  likely  to  pay  to  approximate  that  reality. 

There  are  literally  hundreds  of  machine  simulations  in  the  Depart¬ 
ment  of  Defense’s  current,  active  inventory.  Most  of  them  are  force- 
structure,  weapon- system,  and  technical— evaluation  models. 

Simulations  and  the  people  ("simulators”)  responsible  for  their 
design,  implementation,  and  operation  are  much  harder  to  control  than 
analytic  models  or  modelers.  Not  only  are  fewer  scientific  standards 
available  to  aid  in  evaluating  a  computer  simulation  than  a  mathemati¬ 
cal  model;  there  is  little  or  no  consensus  among  professionals  as  to 
what  the  proper  standards  are. 

Promotion  may  well  be  the  undoing  of  this  potentially  highly  use¬ 
ful  aspect  of  the  profession.  Large-scale  computer  simulations  have 
been  rather  easy  to  sell.  They  appear  reasonable,  and  they  provide  a 
wealth  of  material  for  impressive,  high-level  briefings.  Advocacy  or 
defense  of  a  given  system  or  point  of  view  can  be  made  to  appear  quite 
scientific. 

Machine  simulation  has  been  oversold  in  the  last  ten  years  or  so, 
and  the  shakedown  is  now  taking  place.  The  process  is  healthy,  albeit 
painful  in  certain  quarters .  Much  has  been  learned  that  contributes 
to  the  professionalization  of  work.  Problems  have  been  delineated,  and 
lines  of  communication  are  being  established.  Still,  practitioners 
must  show  more  care  in  (1)  modeling  and  specification,  (2)  data  collec¬ 
tion  and  validation,  (3)  sensitivity  analysis  and  question  formulation, 
and  (4)  accounting  for  costs  and  utilization.  Each  of  these  tasks  is 
a  challenge  by  itself.  Together,  they  make  the  difference  between  a 
viable,  useful  simulation  and  something  that  is  only  useful  to  generate 
visual  aids  for  high-level  "dog  and  pony  show  briefings. 

Man-Machine  Games 

Man-machine  exercises  usually  involve  a  digital  computer  and  peo¬ 
ple  playing  some  of  the  roles  in  the  modeled  system.  People  may  be 
used  merely  because  they  are  cheaper  than  the  software.  Or,  people  may 


be  used  because  human  factors  (particularly  judgment)  are  important  in 
the  situation  being  analyzed. 

In  man -machine  gaming  and  simulation,  the  personnel  are  usually 
analysts,  systems  engineers,  experimental  psychologists,  social  psy¬ 
chologists,  and  economists.  In  work  emphasizing  human  factors,  humans 
are  studied,  not  merely  used  as  inputs.  With  a  systems-engineering  em¬ 
phasis,  humans  may  be  used  because  they  are  handy  and  relatively  cheap. 
In  that  case,  the  orientation  is  more  toward  operations  research  analy¬ 
sis  and  systems  engineering  than  it  is  to  experimental  or  social  psy¬ 
chology. 

The  former  Logistics  Simulation  Laboratory  at  The  Rand  Corporation 

is  an  example  of  a  man-machine  simulation  in  which  people  were  used 

more  as  an  integral  part  of  the  machinery  than  as  subjects  for  human- 
* 

factor  analyses.  John  L.  Kennedy’s  early  work  at  Rand,  on  the  other 

*f* 

hand,  is  an  example  of  human-factors  analysis. 

Man-machine  gaming  and  simulation  was ,  at  its  peak  of  activity 
five  to  seven  years  ago,  probably  oversold.  Currently  it  is  undersold, 
if  not  totally  neglected.  What  is  needed  is  a  basic  inventory  of  in¬ 
formation  that  stresses  substantive  content.  The  connection  between 
machine-only  simulations,  human-factors  analyses,  and  analytic  models 
is  slight,  to  their  mutual  detriment. 

Free-Form  Gaming 

Free-form  gaming  involves  teams  and  a  referee  group  operating 
within  the  framework  of  a  scenario.  If  computational  equipment  is  used 
at  all,  it  is  often  relegated  to  a  simple  bookkeeping  role.  Of  the 
four  classes,  this  one  is  the  least  amenable  to  tight  technical  con¬ 
trol.  It  is,  however,  the  most  likely  to  produce  an  impressive  array 
of  new  insights  into  complex  problems. 

*M.  A.  Geisler  and  A.  S.  Ginsberg,  Man-Machine  Simulation  Experi¬ 
ence,  The  Rand  Corporation,  P-3214,  August  1965. 

+R.  L.  Chapman  and  J.  L.  Kennedy,  The  Background  and  Implications 
of  the  System  Research  Laboratory  Studies,  The  Rand  Corporation,  P-740, 
1955.  See  also  N.  Frederickson,  "Factors  in  In-Basket  Performance," 
Psychological  Monographs,  No.  22,  1962,  for  another  good  example  of  the 
genre . 
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Free-form  gaining  is  also  the  least  expensive.  In  fact,  compared 
with  the  expenses  associated  with  machine  simulations  and  models,  the 
costs  of  all  other  types  seem  minuscule.  A  familiar  example  of  free¬ 
form  gaming  will  suggest  why  and  the  way  in  which  it  is  least  costly. 

The  Political-Military  Exercises  at  and  for  the  Joint  Chiefs  of 
Staff's  Studies,  Analysis,  and  Gaming  Agency  (SAGA)  are  aimed^at  the 
strategic  level  and  involve  relatively  high-priced  personnel.  Strictly 
speaking,  the  cost  of  a  game  or  simulation  must  include  not  only  the 
price  of  the  physical  resources  but  also  the  value  of  the  time  of  the 
personnel  playing  the  game.  In  fact,  the  costs  depend  on  how  one  eval¬ 
uates  the  worth  of  the  time  of  the  top-echelon  teams.  But  if  personnel 
(and  opportunity)  costs  are  ignored,  as  is  usual,  all  that  a  top-level 
political-military  exercise  needs  are  a  couple  of  rooms,  a  few  people 
(who  are  already  in  the  Pentagon),  some  pencils,  pads  of  paper,  and  a 
bit  of  videotape.  Participants  in  the  SAGA  games  may  be  Cabinet  offi¬ 
cials,  three-  or  four-star  generals,  or  admirals,  yet  the  amount  of 
money  involved  for  the  other  resources  is  seldom  more  than  $10,000  per 
game. 

Though  free-form  gaining  is  the  least  tightly  controlled  and  the 
least  expensive  type,  it  receives  far  and  away  the  most  publicity  and 
is  done  at  the  highest  policy  level  of  all  four  types.  Free-form  gam¬ 
ing  also  has  a  few  good  practitioners  and  a  product  that  is  very  hard 
to  measure,  making  it  extremely  difficult  to  ascertain  whether  the  art 
form  has  improved  in  the  last  few  years.  A  little  more  money  and  some 
careful,  rigorous  work  may  yield  some  useful  and  productive  results. 

NONMILITARY  APPLICATIONS  OF  GAMING  AND  SIMULATION 

Before  discussing  our  survey's  findings  about  the  military  uses  of 
models,  simulations,  and  games,  some  observations  on  their  general  uses 
are  in  order.  These  remarks  are  limited  to  simulations  or  models  that 
relate  to  competitive  or  cooperative  behavior,  that  is,  the  behavior 
of  more  than  one  decisionmaker.  Accordingly,  studies  such  as  the 


*For  a  clear  introduction  to  this  type  of  game,  see  Herbert  ^ 
Goldhamer  and  Hans  Speier,  "Some  Observations  on  Political  Gaming, 
Wovld  Politics ,  Vol.  12,  1959,  pp.  71-83. 
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simulation  of  an  oil  refinery  or  transportation  system  are  ruled  out. 
Relevant  applications  fall  into  four  main  classes:  industrial-opera¬ 
tional,  teaching  and  training,  political-diplomatic-military,  and  re¬ 
search. 

Industrial-Operational 

We  have  found  little  use  of  gaming  and  simulation  involving  more 
than  one  decisionmaker  in  any  civilian  setting — in  contrast  to  what  arti¬ 
cles  in  Business  Week,  Fortune,  and  Harvard  Business  Review  suggest. 

While  business  games  are  frequently  used  in  training  programs,  as  enter¬ 
tainment  for  executives,  and  as  teaching  devices  at  business  schools, 
the  production  and  use  of  operational  games  by  industry  appear  to  be 
minimal.  Gaming  for  teaching  purposes  is  particularly  active:  there 
exist  about  500  business  games,  whose  costs  vary  widely  from  a  few  thou¬ 
sand  to  hundreds  of  thousands  of  dollars.  However,  because  of  inade¬ 
quate  accounting,  one  cannot  estimate  the  total  investment  or  even  the 
current  operating  costs  for  such  activities.^ 

i 

Teaching  and  Training 

Recently  there  has  been  a  spurt  in  the  use  of  games  for  introduc¬ 
tory  teaching.  At  the  level  of  university  and  adult  education,  the 
predominant  type  of  game  is  the  business  game  and  its  variants,  which 
are  usually  computerized  games.  One  class  of  such  games  relates  to 
planning  and  development,  including  the  land-use  planning  games  CLUG, 
CITY,  and  METRO,  and  the  air  pollution  exercise  APEX.  Intended  to 
teach  the  management  of  urban  development  problems,  these  games  have 
cost  hundreds  of  thousands  of  dollars  in  total.  The  proliferation  of 
games  concerned  with  social  problems  such  as  congestion,  pollution,  and 
crime  seems  inevitable.  Current  funding  for  these  activities  is  rela¬ 
tively  modest,  but  the  number  and  diversity  of  research  proposals  are 
growing. 

*Such  activities  doubtless  exist  as  strictly  proprietary  ventures. 
The  full  scope  and  magnitude  are,  because  of  secrecy,  probably  unknow¬ 
able. 

+See  Martin  Shubik,  "Gaming:  Costs  and  Facilities, "  Management 
Science,  Vol.  14,  No.  11,  1968,  pp.  629-660. 
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Political-Diplomatic-Military 

Another  type  of  gaining  activity  at  the  university  or  adult-educa¬ 
tion  level  is  political-diplomatic-military  gaming.  It  is  typically 
of  the  free-form  variety  and  relatively  inexpensive.  An  upper  bound 
on  expenditures  during  the  last  20  years  is  certainly  no  more  than 
$2  million  per  annum,  and  probably  much  less.  That  crude  estimate  is 
guessed  at  from  the  budget  of  the  political-military  branch  of  SAGA, 
plus  the  budget  of  Harold  Guetzkow’s  work  on  the  Inter-Nation  Simula¬ 
tion,  plus  a  rough  sum  to  account  for  various  smaller  activities  such 
as  the  work  by  Lincoln  Bloomfield  at  M.I.T.,  and  political-diplomatic- 
military  gaming  at  the  war  colleges,  military  academies,  Rand,  the  Re¬ 
search  Analysis  Corporation,  and  a  few  universities.  The  amount  of 
publicity  given  free-form,  political-diplomatic-military  games  has  been 
enormously  disproportionate  to  the  financial  and  intellectual  invest¬ 
ments  in  them.  Popular  accounts,  such  as  Andrew  Wilson’s  recent  The 
Bomb  ccnd  the  Computer  aside,  research  on  the  intellectual  foundations 
and  uses  of  this  type  of  work  has  been  negligible. 

Research 

Gaming  and  competitive  simulations  devoted  to  research  are  concen¬ 
trated  in  the  universities,  are  associated  with  work  in  social  psychol¬ 
ogy  and  to  some  extent  with  the  formal  theory  of  games,  and  are  slowly 
increasing.  Depending  upon  how  the  research  is  classified  and  costed, 
$2-5  million  a  year  is  probably  being  spent  on  these  activities.  Ex¬ 
amples  are  the  work  of  Anatol  Rapoport  and  his  associates,  the  work  of 
Gerald  Shure  and  Harold  Kelley  at  UCLA,  and  the  activities  of  the  gam¬ 
ing  laboratories  at  Berkeley,  Purdue,  Yale,  Ohio  State,  and  other  uni¬ 
versities.  Specialized  research  agencies  have  used  gaming  as  an  adjunct 
to  research.  An  important  result  of  many  of  these  games  is  the  iden¬ 
tification  of  key  problems  that  are  subsequently  researched  by  other 
methods.  A  small  amount  of  basic  human-factors  analysis  is  being  per¬ 
formed  by  a  few  groups  such  as  the  Army’s  Behavioral  Sciences  Research 
Laboratories . 

*  Andrew  Wilson,  The  Bomb  ccnd  the  Computer:  Wargaming  from  Ancient 
Chinese  Mapboard  to  Atomic  Computer ,  New  York:  Delacorte  Press,  1968. 
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One  could  assign  a  certain  percentage  of  the  budgets  for  activities 
in  the  foregoing  classes  to  research.  However,  it  is  practically  im¬ 
possible  to  figure  the  appropriate  percentage,  and  there  are  strong  in¬ 
dications  that  such  research  is  not  being  undertaken. 

Game  Theory 

One  further  topic  is  the  role  of  mathematical,  game- theory  model 

construction  in  the  study  of  conflict.  In  the  United  States  a  handful 

of  professionals  have  specialized  in  the  investigation  of  duels  and  al- 

* 

location  and  search  games.  This  work  is  a  mathematical  art  form  that 
has  added  to  the  basic  knowledge  about  competitive  situations  at  a  rela¬ 
tively  low  cost.  These  few  professionals  require  no  extraordinary  equip¬ 
ment;  hence,  keeping  them  fully  employed  amounts  to  providing  salaries 
and  standard  overheads.  A  liberal  upper-bound  estimate  on  total  expendi¬ 
tures  for  such  game- theoretic  analyses  of  conflict  situations  is  $1-2 
million  a  year. 

Summary 

The  major  activities  and  expenditures  in  operational  gaming  and 
simulation  to  study  conflict  and  cooperation  are  by  the  military.  Such 
expenditures  elsewhere  are  negligible.  Expenditures  for  teaching¬ 
training  gaming-  for  university  and  adult  education  amount  to  no  more 
than  a  few  million  dollars  but  are  undoubtedly  growing.  Gaming  in  in¬ 
troductory  education  appears  to  be  expanding  as  well.  In  research,  ex¬ 
perimental  activity  at  the  universities  is  increasing,  but  except  for 
a  few  corporate-sponsored  projects  at  places  such  as  Rand  and  RAC,  there 
appears  to  be  very  little  direct  funding  of  basic  research  elsewhere. 

That  includes  the  military,  except  for  a  small  amount  of  human-factors 
analysis . 

Publicity  devoted  to  gaming  and  simulation  appears  to  vary  inversely 
with  actual  expenditures  and  activities.  High-level  political-military 
exercises  may  have  caught  the  imagination  of  the  public,  but  they  have 


M.  Dresher,  J.  Mayberry,  S.  Moglewer,  L.  Shapley,  and  several 
others  come  to  mind. 


not  caught  even  a  minute  fraction  of  the  total  resources  being  devoted 
to  gaming  and  simulation. 

Professional  interest  and  activity  in  gaming  and  competitive  sim¬ 
ulation  is  reflected  in  the  membership  of  the  Military  Operations  Re¬ 
search  Society  (MORS),  which  has  some  4400  members  on  its  current  mail¬ 
ing  list  (down  from  a  peak  of  about  5500  several  years  ago) ,  and  of  the 
Institute  of  Management  Science's  College  on  Gaming  and  Simulation, 
which  had  a  1971  membership  of  around  650  (slightly  larger  than  several 

years  ago) . 

The  growing  literature  is  another  indicator  of  activity.  Of  course, 
it  shows  a  pronounced  bias  toward  the  nonmilitary  aspects  of  gaming  and 
simulation  because  military  studies,  even  if  they  are  not  classified, 
normally  do  not  appear  in  the  published  literature.  Of  the  approxi¬ 
mately  2000  articles  and  books  that  were  investigated  in  the  course  of 
the  authors’  recent  research,  about  15  percent  were  operational  and  less 

than  12  percent  were  devoted  to  war  gaming  and  political-diplomatic- 

* 

military  gaming. 

Historically  it  appears  that  activity  and  expenditures  on  gaming 
and  competitive  simulation  hit  a  peak  in  the  early  to  middle  1960s  and 
have  been  on  a  gentle  decline  since  then. 


*A  survey  and  evaluation  of  the  literature  are  contained  in  Martin 
Shubik,  Garry  D.  Brewer,  and  Eve  Savage,  The  Literature  of  Garmng ,  Sim¬ 
ulation,  and  Model-Building:  Index  and  Critical  Abstracts,  The  Rand 
Corporation,  R-620-ARPA,  1972,  and  Martin  Shubik  and  Garry  D.  Brewer, 
Reviews  of  Selected  Books  and  Articles  on  Garmng  and  Svmlatvon,  The 
Rand  Corporation,  R-732-ARPA,  1972. 
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II.  THE  SURVEY  AND  ITS  RESULTS 


Having  ascertained  that  the  major  expenditures  are  devoted  to  com¬ 
petitive,  all-machine  simulations  and  models,  considerably  less  to  man- 
machine  games,  and  the  least  amounts  to  free— form  games,  we  identified 
approximately  450  active  military  models,  simulations,  and  games  of 
those  types  in  several  Department  of  Defense  and  service  catalogs  and 
inventories .  In  close  collaboration  with  the  General  Accounting  Office, 
which  had  independently  been  asked  by  the  House  Appropriations  Committee 
to  assess  the  kind,  extent,  and  use  of  military  gaming,  we  initially 
chose  a  sample  of  150  models,  simulations,  and  games,  which  number  was 
later  pared  down  to  135.  A  detailed,  70-page  questionnaire  was  circu¬ 
lated  to  the  135  groups,  agencies,  or  individuals  in  DOD  that  were 
judged  to  be  most  knowledgeable  about  each  model,  simulation,  or  game. 

We  received  133  replies,  one  of  which  gave  so  little  information  that 
it  was  unusable.  Thus,  unless  otherwise  noted,  the  statistical  ma¬ 
terial  below  is  based  on  a  respondent  population  of  132.  Replies  were 
coded  for  computerization  (see  the  codebook,  Appendix  C) ,  with  multiple 
checks  on  the  accuracy  of  transcription. 

The  sample  was  chosen  to  include  several  of  the  larger  or  more 
heavily  used  simulations  and  games.  The  sample  was  also  biased  somewhat 
towards  the  Army,  there  being  59  Army,  26  Air  Force,  35  Navy,  and  11 
other  DOD  personnel  in  the  population.  Extrapolations  from  this  sample 
to  the  total  450  models,  simulations,  and  games  should  be  made  carefully, 
keeping  these  biases  in  mind, 

* Three  preliminary  versions  of  the  questionnaire  were  developed  and 
tried  out  on  known  games,  utilizing  cooperative  and  competent  respondents. 
Our  thanks  to  E.  W.  Paxson,  J.  R.  Lind,  H.  Guetzkow,  M.  G.  Weiner,  and  A. 
Theony  for  their  constructive  assistance  with  this  critical  portion  of 
the  research.  The  final  version  of  the  questionnaire  was  published  as 
Martin  Shubik  and  Garry  D.  Brewer,  Questionnaire:  Models ,  Computer  Ma¬ 
chine  Simulations 3  Games  and  Studies 3  The  Rand  Corporation,  P-4672,  July 
1971.  It  is  attached  as  Appendix  A,  filled  out  with  summary  results  of 
the  survey,  and  serves  as  an  organizing  device  for  the  body  of  this  re¬ 
port. 

^Appendix  B  provides  a  full  list  of  the  names  of  the  models,  simu¬ 
lations,  and  games  surveyed  in  the  preliminary  and  final  stages  of  this 
project. 
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Total  reported  development  costs  were  approximately  $32  million, 
but  approximately  30  responses  did  not  provide  appropriate  or  clear 
cost  information.  They  included  several  of  what  are  believed  to  be 
the  larger  simulations.  Thus,  by  simple  averaging,  total  costs  approx¬ 
imate  $40  million.  Since  the  sample  represents  around  30  percent  of 
the  total  inventory,  a  crude  estimate  of  total  investment  is  $130-$140 
million  for  all  active  models,  simulations,  and  games  in  the  DOD  inven¬ 
tory,  Using  other  costing  criteria  before  our  questionnaire  was  circu¬ 
lated,  the  General  Accounting  Office  made  a  preliminary  estimate  of 
$170.5  million,  which  seems  a  plausible  upper  bound.  On  the  average, 
the  inventory  represents  a  three-  to  four-year  supply  (an  average  model, 
simulation,  or  game  "lasts"  three  to  four  years) ;  hence  we  estimate  that 
$30  to  $40  million  per  year  has  been  expended  recently  for  construction 
costs.  These  estimates  are  of  necessity  very  rough.  In  fact,  one  of 
our  major  findings  concerns  the  poor  to  nonexistent  cost-accounting 
definitions  and  procedures. 

The  size  of  expenditures  and  number  of  activities  do  not  provide 
all  the  clues  to  the  importance  of  the  work.  Many  other  criteria  are 
needed  to  judge  these  activities.  But  valid  criteria  have  not  yet  been 
formulated  for  the  profession.  With  these  caveats  in  mind,  let  us  turn 
to  a  broad  description  of  the  survey  results. 

THE  RESPONDENTS 

When  the  final  version  of  the  questionnaire  was  set,  the  General 
Accounting  Office  met  with  personnel  at  DOD  "points  of  contact,"  de¬ 
scribed  the  rationale  of  the  survey,  and  asked  that  the  most  knowledge¬ 
able  person  about  any  given  model,  simulation,  or  game  be  assigned  to 
answer  each  questionnaire.  If  one  person  could  not  do  it,  groups  or 
committees  were  acceptable  substitutes  as  long  as  the  responses  were 
the  most  up-to-date  and  accurate  possible. 

As  it  turned  out,  most  questionnaires  were  filled  out  by  one  person, 
and  most  respondents  filled  out  only  one  questionnaire.  In  a  few 

U.S.  General  Accounting  Office,  Computer  Simulations >  War  Gaming 
and  Contract  Studies y  A  Report  to  the  Committee  on  Appropriations, 

House  of  Representatives,  Washington,  D.C.,  February  23,  1971,  p.  8. 
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instances,  one  man  was  responsible  for  two  or  perhaps  three  separate 
entries.  One  respondent  did  yeoman  service  on  six  or  seven  question¬ 
naires  . 

Respondents  were  mostly  users,  42  or  31,8  percent,  or  designer/ 

builders,  41  or  31.1  percent.  The  next  largest  classes  of  respondents 

were  funder /sponsors ,  19  or  14.4  percent,  and  caretakers,  13  or  13.6 
* 

percent. 

The  average  length  of  time  required  to  fill  out  the  questionnaire 
was  10.5  hours.  The  modal  time  was  4  hours,  and  84  were  able  to  answer 
in  8  hours  or  less;  however,  two  or  three  respondents  stated  that  they 
took  as  much  as  70  hours. ^ 

One  might  thus  presume  that  this  group  of  respondents  would  be 
able  to  provide,  as  well  as  or  better  than  any  other  group  that  might 
be  formed,  reasoned  and  valuable  replies  to  the  questionnaire.  The 
following  pages  discuss  their  replies,  grouped  under  the  headings  pur¬ 
poses,  production,  operations,  use,  and  costs  of  models,  simulations, 
and  games,  and  their  opinions  about  certain  hypothetical  innovations 
in  the  gaming  profession. 

PURPOSES 

Because  there  is  so  little  agreement  even  about  fundamental  defi¬ 
nitions  ,  it  seemed  especially  important  to  find  out  what  terms  and  con¬ 
cepts  the  professional  community  is  currently  using  to  describe  the 
rationale  or  intent  of  various  models,  simulations,  and  games.  The 
intention  or  purpose  for  which  an  MSG  is  built  has  a  direct  and  im¬ 
portant  bearing  on  how  one  subsequently  goes  about  specifying,  vali¬ 
dating,  and  controlling  it. 

*The  name,  affiliation,  and  address  of  each  respondent  have  been 
coded  and  included  in  the  data;  however,  that  information  is  not  re¬ 
ported  here. 

^Reported  time  to  complete  each  questionnaire  is  listed  in  Appen¬ 
dix  B. 

^For  convenience,  the  authors  have  expressed  the  notion  of  model, 
simulation,  or  game  as  ,,MSG,,,  which  term  is  used  hereinafter. 
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Basic  Categories 

We  began  by  asking  each  respondent  to  identify  his  MSG  as  either 
a  model,  simulation,  a  man-machine  or  manual  game,  a  mathematical  anal¬ 
ysis  or  study,  or,  if  none  of  those  categories  fit,  as  an  "other."  Re¬ 
sults  are  shown  in  Fig.  1.  Most  were  identified  as  models.  The  usage 
of  that  term  and  concept  is  vague,  however;  one  respondent  noted  that 
to  him  a  model  meant  any  regression  equation,  therefore  the  universe 
of  models  is  so  large  as  to  be  virtually  unbounded.  In  short,  the 
word  has  little  referential  utility.  The  terminological  problem  is  un¬ 
derscored  by  the  General  Accounting  Office’s  initial  lack  of  success 
when  it  asked  the  military  services  to  provide  data  on  "war  games."  The 
GAO  found  that  practically  no  war  games  are  played;  rather,  models  and 
simulations  are  built. 


Fig.l — Respondents1  primary  classification  of  their  MSGs 
(variable  CATEG1)* 


*See  Appendixes  A  and  C  for  complete  definitions  of  all  variables 
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Another  striking  finding  is  how  few  man-machine  and  manual  exer¬ 
cises  are  in  the  DOD’s  active  inventory  (14  or  10.6  percent).  While 
our  sample  represents  132  of  the  some  450  active  MSGs,  we  are  reason¬ 
ably  confident  that  10.6  percent  is  an  accurate  estimate  of  the  current 
low  level  of  man-machine  and  manual  (or  free-form)  activity. 

Multiple  choices  were  allowed;  that  is,  a  respondent  could  call 
his  MSG  both,  say,  a  model  and  a  man-machine  exercise.  However,  only 
26  indicated  a  secondary  choice,  and  of  that  subset,  19  chose  simula¬ 
tion,  3  "other,"  and  2  each  models  or  mathematical  analysis.  No  one 
selected  as  a  secondary  choice  man-machine  or  manual  game. 

Concepts  and  Questions 

The  important  concept  of  purpose  is  inherently  complex.  Accord¬ 
ingly,  our  probings  were  detailed  and  intentionally  redundant.  Re¬ 
spondents  were  asked  to  describe  in  their  own  words  the  major  stated 
purpose  of  their  MSG  and  to  cite  two  specific  questions  or  operational 
problems  that  the  MSG  had  been  used  to  answer.  Next,  they  were  asked 
to  check  off  appropriate  characterizations  of  purpose  from  a  list  of 
eight  categories,  one  of  which  was  "other — specify."  We  also  asked 
them  to  indicate  their  confidence  in  that  response,  from  low  to  abso¬ 
lutely  certain.  Several  questions  later,  purpose  was  brought  up  again, 
with  reference  to  who  had  initiated  the  work,  what  he  wanted,  and  how 
well  he  specified  his  wishes.  And  finally,  several  questions  later,  we 
asked  simple  yes-no  questions  about  whether  the  MSG  was  initially  de¬ 
signed  for  experimental  or  educational  purposes.  The  results  of  this 
repeated  probing  are  revealing,  especially  when  tabulated  with  other 
descriptive  categories  such  as  who  paid  for,  built,  or  used  the  MSG, 
at  what  cost,  and  so  forth. 

Results 

Description.  Primary  purposes  are  shown  in  Fig.  2.  No  respondent 

For  a  theoretical  examination  of  purposes,  see  Martin  Shubik,  On 
the  Scope  of  Gaming ,  The  Rand  Corporation,  P-4608,  March  1971,  repub¬ 
lished  in  Management  Science3  Vol.  18,  No.  5,  January  1972,  Part  2, 
pp.  P-20-P-36. 
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Fig.  2 —  Primary  stated  purpose  of  MSG  (variable  PURP  1) 

indicated  that  his  MSG  was  intended  primarily  for  military-political- 
economic  and  training/education  purposes.  In  fact,  only  one  MSG  was 
listed  under  political— diplomatic— military  (abbreviated  PDM) /interna¬ 
tional  relations.  Given  present-day  concern  over  strategic  arms  limi¬ 
tation,  perpetual  Middle  East  tensions,  and  the  expansion  of  Sino- 
American  relations,  that  is  a  startling  finding.  Even  when  secondary 
and  tertiary  purposes  are  taken  into  account,  the  findings  are  not  much 
different.  Some  61  or  46.2  percent  of  the  MSGs  had  only  one  purpose, 
and  of  the  remainder,  one  each  had  secondary  purposes  of  PDM/interna- 
tional  relations  and  training/education.  Of  the  MSGs  with  three  pur¬ 
poses  (36  or  27.3  percent),  two  were  military-political-economic  and 
one  each  was  PDM/ international  relations  and  training/education.  Re¬ 
search/methodology  received  similar  scant  attention.  It  was  the  pri¬ 
mary  purpose  of  2  MSGs,  the  secondary  purpose  of  5,  and  the  tertiary 
purpose  of  10. 

By  far  the  greatest  emphasis  in  current  DOD  activity  is  on  all 
computer,  technical  evaluations  (59  or  44.7  percent),  force-structure 
analysis  (38  or  28.8  percent),  and  doctrinal  evaluation  (24  or  18.2 
percent).  When  it  comes  to  secondary  purpose,  the  mix  changes.  Of  the 
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71  MSGs  with  more  than  one  stated  purpose,  32  were  intended  to  study 
doctrine,  28  for  force  structure,  and  only  one  for  technical  evaluation. 
At  the  tertiary  level,  with  36  reported,  17  were  for  force-structure 
analysis,  10  were  for  research  and  methodological  development,  and  the 
remainder  were  sprinkled  among  other  purposes. 

Technical  evaluation  turns  out  to  mean  weapon  system  evaluation 
when  one  reads  the  written  accounts  of  each  MSG’s  major  stated  purpose 
and  examples  of  use.  Such  activities  are  appropriate  enough,  but  only 
if  the  data  are  valid,  are  available  for  scrutiny  by  responsible  per¬ 
sons,  and  if  scientific  criteria  and  procedures  such  as  replication, 
external  review,  and  documentation  are  followed.  If  such  rigor  is  not 
assumed,  one  suspects  that  much  effort  is  going  into  building  uneval¬ 
uated  MSGs  that  support  specialized  points  of  view.  The  scarcity  of 
attention  given  research,  training,  and  more-or— less  political  mat 
ters  could  lead  one  to  believe  that  the  DOD  is  modeling  problems  that 
are  easily  quantified  and  are  well  enough  understood  that  no  new  theo¬ 
retical  research  is  needed  to  explain  them.  If  so,  then  either  the 
problems  confronting  the  DOD  are  in  fact  being  managed  in  a  rigorous, 
scientific  fashion,  or,  as  appears  to  be  the  case,  intangible,  intract¬ 
able,  or  "soft"  issues  are  largely  being  overlooked. 

Respondents  were  generally  highly  confident  or  absolutely  certain 
about  their  MSG’s  purpose  (105  or  79.5  percent).  However,  27  or  20.5 
percent  either  had  low  to  moderate  confidence  in  their  answers  or  did 
not  respond  at  all.  Considering  the  composition  of  the  respondent 
population  and  the  inquiry’s  intent  and  sponsorship,  that  figure  seems 
inordinately  high.  If  these  people  do  not  know  what  their  models  are 
supposed  to  do,  who  does? 

Initiators  of  MSG  development,  as  shown  in  Fig.  3,  strongly  favored 
analysis  and  diagnosis  as  a  primary  purpose  (108  or  81.8  percent).  Far 
fewer  favored  operations  (14),  research  and  theory  development  (7),  ex¬ 
perimentation  (1) ,  or  teaching-training  (2) .  That  finding  is  consistent 


Confidence  levels,  the  degree  of  the  respondent  s  certainty  about 
an  answer,  were  required  on  most  questions.  The  categories  ranged  over 
low,  moderate,  high,  to  absolutely  certain,  assigned  values  1  through 
4,  respectively. 
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Number  of  respondents 


Fig.  3 — MSG  initiators  primary  purpose  (variable  INPURP1) 

with  the  heavy  emphasis  on  technical  evaluation  and  force-structure 
analysis  already  observed.  When  asked  how  tightly  funding  sources 
specified  their  intentions  or  purposes,  77  respondents  replied  with 
either  high  or  absolute  certainty  that  33  or  25.0  percent  of  the  sample 
were  tightly  specified,  that  is,  the  funding  source  defined  well  what 
it  wanted  for  its  money.  In  26  or  19.7  percent  of  the  cases,  however, 
the  respondent  either  did  not  know  or  did  not  respond  to  the  question. 
Those  replies  are  shown  in  Fig.  4. 


Number  of  respondents 


Fig  .4 — Specificity  of  initiator’s  purpose  (variable  SPCPRP) 
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A  full  114  or  86.4  percent  of  the  MSGs  were  not  intended  for  ex¬ 
perimental  purposes,  and  121  or  91.7  percent  were  not  intended  to  be 
used  for  educational  purposes.  The  sample  may  be  tentatively  charac¬ 
terized  as  operational  machine  models  and  simulations  used  for  techni¬ 
cal  evaluations  and  force-structure  analyses.  Research,  experimenta¬ 
tion,  and  training  and  education  are  all  of  considerably  less  importance 
in  this  sample. 

Trends .  Fundamental  purposes  have  shifted  somewhat  for  MSGs  ini¬ 
tiated  since  1966.  Of  all  MSGs  produced,  technical  evaluations  have 
declined  somewhat,  from  50.0  percent  for  1966-67  to  38.9  percent  for 
1971-72;  force-structure  analyses  have  increased  from  13.6  percent  to 
38.9  percent;  and  doctrine  evaluations  have  remained  rather  steady  at 
about  22.0  percent  of  the  total. 

Since  1966,  not  a  single  one  of  these  MSGs  was  initiated  for 
teaching  and  training  purposes.  Analysis  and  diagnosis  as  a  purpose 
has  held  steady  at  about  85  percent  of  the  total  initiated. 

Relationship  to  Other  Descriptors.  Respondents  knew  in  general 

what  their  MSGs  were  supposed  to  do,  as  indicated  when  primary  purpose 

was  cross- tabulated  against  the  respondent’s  confidence  in  his  reply. 

Some  89.5  percent  were  either  highly  or  absolutely  certain  about  purpose; 

however,  10  of  the  38  force-structure  MSG  respondents  indicated  low  or 

2 

middle  confidence  or  did  not  bother  to  answer  at  all  [X  p  <  .024], 

The  reason  is  not  clear,  but  a  possible  explanation  is  that  force-struc¬ 
ture  MSGs  are  not  well  documented  or  are  "one-shot"  affairs  whose 
initial  purpose  is  no  longer  remembered  by  the  organizations  responsible 
for  them.  We  shall  reexamine  that  hypothesis  later. 

Looking  at  the  MSG  funding  source  indicates  that  the  Navy  concen¬ 
trates  heavily  on  technical  evaluation,  26  out  of  33  or  78.8  percent; 
the  Air  Force  divides  between  force-structure  analysis,  12/23  or  52.2 
percent,  and  technical  evaluation,  10/23  or  43.5  percent;  the  Army  is 
evenly  divided  between  technical  evaluation,  18/56  or  32.1  percent, 
doctrine  evaluation,  16/56  or  28.6  percent,  and  force-structure  analysis, 
16/56  or  28.6  percent;  and  5  of  the  Joint  Chiefs  of  Staff’s  entries 
were  on  force  structure  and  one  was  on  PDM/ international  relations 
[X  p  <  .001].  No  direct  Department  of  State  or  National  Security  Council 
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funding  was  noted,  reinforcing  an  earlier  observation  about  the  sample’s 
apparent  dearth  of  explicitly  political  or  diplomatic  substance. 

MSG  initiator  cross-tabulated  with  purpose  requires  some  explana¬ 
tion.  Model-builders  or  researchers  may  as  individuals  propose  MSG 
construction  to  an  appropriate  funding  source.  That  differs  from  in- 
house  work,  which  is  limited  either  by  the  eventual  users  or  by  a  non¬ 
user  management  source  acting  for  the  users.  An  agency  outside  an  or¬ 
ganization  may  request  an  organization  to  build  an  MSG  for  its  use,  or 
an  agency  may  request  a  different  organization  both  to  build  and  use 
an  MSG  for  the  initiator’s  own  purposes.  Individual  researchers  ini¬ 
tiated  24  or  18.2  percent  of  all  132  respondent  MSGs,  and  of  those  there 
was  a  fairly  even  division  between  technical,  doctrinal,  and  force-struc¬ 
ture  evaluation,  and  analytical  purposes.  Of  the  40  MSGs  initiated  by 
in-house  users,  18  or  45.0  percent  were  for  technical  evaluation.  That, 
plus  those  initiated  by  users  external  to  the  building  agency,  accounts 
for  34/59  or  57.6  percent  of  all  the  technical-evaluation  MSGs.  Thus, 
it  appears  that  users  tend  to  initiate  their  own  technical-evaluation 
MSGs  [X2  p  <  .011]. 

The  initiator’s  purpose,  whether  teaching,  analysis,  operations, 
experimentation,  or  research,  when  tabulated  against  the  MSG’s  purpose 
or  category  of  intended  use,  shows  that  the  penchant  to  do  analysis 
and  diagnosis  runs  evenly  through  all  MSG  purposes.  For  technical 
evaluation,  50/59  or  84.9  percent  are  intended  for  analysis  and  diag¬ 
nosis;  for  doctrine  evaluation,  it  is  21/24  or  87.5  percent;  and  for 

2 

force  structure,  it  is  30/38  or  78.9  percent  [X  p  <  .001].  The  one 
surprising  finding  is  that  when  an  initiator  intended  research  and 
theory  development  (7/132  or  5.3  percent),  it  resulted  in  MSGs  whose 
primary  purpose  was  technical  evaluation  (4/7)  ,  force-structure  analy¬ 
sis  (2/7),  or  doctrine  evaluation  (1/7),  not  the  development  of  research 
or  methodology  [X  p  <  .001].  It  appears  that  even  when  one  has  a  seri¬ 
ous  research  problem,  the  resulting  MSG  is  not  reported  as  being  pri¬ 
marily  geared  to  research.  As  for  the  two  MSGs  whose  prime  purpose  was 

listed  as  research  and  methodology,  initiators  intended  analysis,  diag- 

2 

nosis,  and  experimentation,  not  research  per  se  IX  p  <  .001]. 
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Two  descriptive  classes  of  information,  the  ease  of  transfer  of 
MSG  operations  from  one  site  to  another  and  the  extent  of  documenta¬ 
tion,  provide  insight  into  the  adequacy  of  scientific  controls* 

Technical-evaluation  MSGs  do  not  transfer  at  all  in  15.3  percent  of 
the  59  cases;  transfer  with  great  expense  in  time  and  technical  talent 
in  22.0  percent  of  the  cases,  and  are  easily  transferable  in  but  8  of 
the  cases,  or  13.6  percent  of  the  time.  The  relationship  between  pur¬ 
pose  and  ease  of  transfer  was  not  statistically  significant,  so  we  are 
unable  to  place  too  much  confidence  in  what  the  descriptor  means.  How¬ 
ever,  the  tabulation  of  purpose  versus  extent  of  documentation  was 
highly  significant  [X  p  <  .001]  and  suggestive.  By  the  respondents 
own  assessment  of  documentation,  which  we  found  to  be  overgenerous 
in  several  cases  where  documentation  was  probed  in  detail,  15  of  the 
technical-evaluation  MSGs  had  either  weak,  poor,  uneven,  unavailable, 
or  unknown  documentation.  Of  the  38  force-structure  MSGs,  9  fell  into 
those  categories.  That  information  is  not  conclusive,  however,  because 
roughly  equal  numbers  in  each  category  responded  that  their  MSG  had 
excellent,  very  good,  or  average  documentation.  While  the  distributions 
are  flat,  the  fact  that  25.4  percent  of  the  technical-evaluation  and 
23.6  percent  of  the  force-structure  MSGs  were  voluntarily  acknowledged 
to  have  weak  or  worse  documentation  seems  important. 

Cost  data  were  broken  down  into  coded  categories  and  were  tabulated 
against  purpose.  Immediately  one  is  struck  by  the  fact  that  25  or  18.9 
percent  of  the  respondents  were  unable  to  supply  any  information  what¬ 
soever  on  costs.  Of  the  remainder,  50  MSGs  cost  $100,000  or  less,  22 
cost  $100,000-$249, 000,  and  17  cost  $250,000-$500,000.  Of  the  18 
MSGs  costing  more  than  $500,000,  6  were  technical  evaluations,  5 

were  doctrine  evaluations,  5  were  force-structure  studies,  and  1  was 

2 

for  political-diplomatic-military  purposes  [X  p  <  .012].  Later  we 
shall  take  up  cost  considerations  in  more  detail. 

PRODUCTION 

Under  "purposes,"  several  basic  questions  were  asked  about  the 
MSG1 s  production.  It  is  an  important  topic  that  warrants  analysis  by 
itself,  so  we  later  posed  more  detailed  questions,  such  as  how  many 
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predecessors  and  spinoffs  are  traceable  to  the  MSG ;  how  long  it  was 
under  development  before  being  used;  how  many  man-years  were  taken  to 
build  it;  and  where  the  data  came  from  to  specify  it.  Another  set  of 
questions  asked  respondents  to  describe  and  characterize  the  MSGs  that 
resulted  from  the  production  process. 

Concepts  and  Questions 

While  some  production  questions,  such  as  what  MSGs  preceded  or  fol¬ 
lowed  the  investigated  MSG,  are  straightforward  and  easily  answered, 
others  are  not. 

When  discussing  development  time,  for  example,  it  is  difficult  to 
be  both  precise  and  accurate.  We  refined  the  question  to  include  the 
elapsed  time  between  the  decision  to  build  a  given  MSG  and  its  first 
production  run,  thereby  grossly  underestimating  in  those  cases  when  an 
MSG  continues  to  be  developed  even  while  it  is  being  used.  Another 
illustration  of  conceptual  difficulties  is  the  estimation  of  human  re¬ 
source  expenditures.  One  must  distinguish  between  total  man-years, 
professional  man-years,  and  programmer  man-years  to  understand  what 
kinds  of  talent  have  been  involved  in  MSG  construction. 

To  understand  the  sample  better,  several  descriptive  questions 
about  scenarios,  mathematical  difficulty,  timing,  levels  of  resolution, 
use  of  random  events,  and  supporting  data  were  asked.  Taken  together, 
they  provide  valuable  insights  into  current  production  practices. 

Results 

Description.  About  three-fourths  of  the  sample  had  at  least  one 
direct  parent  or  antecedent  MSG,  indicating  the  cumulative,  continuous 
nature  of  much  of  the  work.  While  the  sample  was  biased  to  include 
many  active  MSGs,  there  were  45  cases  where  a  spinoff  or  distinct  prog¬ 
eny  was  reported  to  have  been  developed.  These  activities  are  clearly 
not  discrete  events  but  represent  a  continuous,  ongoing  process  of  ini¬ 
tiation,  production,  and  use. 

Alternative  procedures  or  methods  to  the  one  chosen  were  elicited 
by  asking  the  respondents  to  imagine  achieving  the  objectives  of  the 
MSG  by  different  means.  As  shown  in  Fig.  5,  analysis  was  the  dominant 
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Number  of  respondents 


Fig.  5 — Best  alternative  method  to  one  used  (variable  ALTPRO  1 ) 

alternative,  reinforcing  an  earlier  observation  on  the  widespread  use 
and  acceptance  of  the  term  "analysis.”  The  fact  that  22.0  percent  of 
the  respondents  believed  that  no  alternatives  existed  supports  the  view 
that  much  of  this  activity  is  of  a  "last-resort"  methodological  variety: 
many  of  the  problems  are  not  tractable  by  other  means. 

MSGs  were  constructed  largely  by  the  armed  forces  themselves,  59 
or  44.7  percent  of  the  entire  sample.  For-profit  organizations  built 
37  or  28.7  percent;  not-for-prof its  contributed  29  or  22.0  percent; 
and  universities  accounted  for  7  or  5.3  percent. 

Development  time,  the  elapsed  time  between  initiation  and  the  first 
production  run,  varies  widely  between  less  than  three  months  to  more 
than  42  months.  The  distribution  is  shown  in  Fig.  6.  Since  63.6  per¬ 
cent  were  developed  in  18  months  or  less,  one  might  conclude  that  de¬ 
cisionmaking  lead  times  are  most  likely  to  run  under  two  years.  It 
took  two  to  five  man-years  to  develop  31.8  percent  of  the  sampled 
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Fig.  6 —  MSG  development:  Elapsed  time  (months)  from  initiation 
to  first  production  run  (variable  DEVTM1) 

MSGs.  Less  dominant  categories  are  depicted  in  Fig.  7.  Professional 
man-years  and  total  man-years  are  nearly  equivalent,  suggesting  that 
most  development  activity  was  carried  out  by  professionals  themselves. 
Indeed,  written  profiles  of  professional  teams  support  this  point 
nicely.  About  one  computer  programmer  man-year  per  MSG  was  the  norm. 

Our  next  set  of  questions  dealt  with  a  characterization  of  the 
MSGs  and  their  data  bases.  A  quick  summary  of  the  findings  revealed 
that  54  percent  used  numerical  scenarios  only.  Judging  from  the  analy¬ 
sis  and  written  replies  to  the  questionnaire,  little  attention  appears 
to  be  paid  to  intangible  assumptions  or  to  qualitative  factors  in  the 
studies . 
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Number  of  respondents 

Fig.  7 — MSG  development:  Human  resource  expenditures 
(man-years)  (variable  DEVMY  1 ) 

In  spite  of  a  popular  belief  that  ,!advanced  mathematical  methods 
are  taking  over,  the  level  of  mathematical  sophistication  required  to 
work  with  the  MSGs  was  regarded  as  high  ("requires  an  advanced  degree 
to  interpret  the  output")  in  only  8  percent  of  the  cases. 

Most  MSGs,  52  percent,  had  as  their  time  setting  either  an  unspeci¬ 
fied  time  or  the  future. 

Of  all  the  MSGs,  approximately  45  percent  were  directed  toward  the 
study  of  combat  at  the  engagement  level,  10  percent  at  the  battle  level, 
11  percent  at  the  campaign  level,  and  14  percent  at  the  level  of  war. 

The  level  of  temporal  resolution  was  rather  small.  Fully  79  or 
66.8  percent  were  detailed  down  to  either  minutes  or  seconds.  Similarly, 
spatial  resolution  was  also  fine-grained;  about  35  percent  were  based 
on  meters,  and  some  27  percent  were  detailed  at  the  kilometer  level. 

The  sample  was  split  about  50/50  on  the  use  of  random  elements. 

We  were  surprised,  given  the  quantitative  and  qualitative  advances 
in  computer  languages,  that  so  many  MSGs  were  still  written  in  FORTRAN 
more  than  80  percent.  The  remainder  were  spread  widely  among  SIMSCRIPT, 
ALGOL,  ASSEMBLY,  COBOL,  GPSS,  and  several  others. 
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On  the  question  of  MSG  size,  we  asked  for  the  approximate  number 
of  computer  instructions  contained  in  the  model  itself,  less  any  data 
inputs.  On  the  average,  they  are  either  very  large  (36  or  27.3  percent 
had  over  10,000  instructions)  or  moderately  small  (24  or  18.2  percent 
had  2,000  or  less);  45  or  44.0  percent  contained  less  than  4,000,  or 
about  two  boxes  of  IBM  cards. 

In  spite  of  our  difficulty  in  getting  information  about  these  MSGs, 
72  percent  were  unclassified  as  long  as  data  input  values  were  not  at— 
tached.  With  input  data,  only  12  percent  were  still  unclassified.  Data 
are  classified  predominantly  at  the  Secret  level.  Thus,  there  is  little 
reason  for  classification  to  prevent  most  of  the  sampled  MSGs,  at  least, 
from  being  made  available  for  scientific  scrutiny. 

There  is  an  indication  that  data-source  and  validity  problems  are 
acute.  Careful  gathering  of  field-test  data  or  experimental  informa¬ 
tion  tends  to  be  expensive  and  time-consuming,  especially  the  prepara¬ 
tion  of  planning  factors;  few  of  these  MSGs  benefited  from  such  measures. 
The  modal  procedure  is  to  have  other  military  agencies  supply  data  with¬ 
out  any  follow-up  or  cross— checking  by  the  user  agency*  In  less  than 
30  percent  of  the  replies  was  there  any  indication  that  additional  ef¬ 
fort  had  been  made  to  check  the  validity  of  the  data. 

Trends .  Of  MSGs  developed  since  1966,  development  time  has  fluc¬ 
tuated  so  that  no  clear  trends  are  apparent.  There  is  a  slight  trend 
toward  fewer  technical  evaluations  and  more  force-structure  analysis; 
doctrine  evaluations  have  been  relatively  stable. 

The  technical  characteristics  of  MSGs  are  all  quite  stable.  That 
is  to  say,  there  are  no  discernible  changes  in  the  proportion  of  nu¬ 
merical  versus  verbal  scenarios  being  used,  in  the  level  of  mathemati¬ 
cal  sophistication  being  "built  into"  the  MSGs,  in  the  level  of  temporal 
resolution,  or  in  the  split  between  those  using  and  those  not  using 
stochastic  or  random  elements. 

Relationship  to  Other  Descriptors.  Of  the  59  MSGs  constructed  in- 
house,  31  were  for  the  purpose  of  technical  evaluation,  17  were  for 
force-structure  analysis,  and  8  were  doctrine  evaluations.  Of  those 
built  by  for-profit  organizations,  11  each  were  technical  evaluations 
and  force-structure  analyses,  and  8  were  doctrine  evaluations.  Not- 
for-profits  built  11/29  or  37.9  percent  for  technical  evaluation,  10/29 
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or  34.5  percent  for  force-structure  analysis,  and  8/29  or  27.6  percent 

for  doctrine  evaluation.  Universities  concentrated  6  of  their  7  MSGs 

on  technical  evaluation.  Of  the  2  research  MSGs,  1  was  done  by  a  uni- 

2 

versity  and  1  by  the  armed  forces  [X  p  <  .007]. 

Multiple  authorship  or  participation  in  the  construction  phase  was 
noted  in  17  instances.  Not-for-prof its  did  11  of  them,  mostly  on  a 
piecemeal  or  consultative  basis,  and  8  of  those  were  for  technical  eval¬ 
uation  purposes.  The  remaining  6  multiple-author  MSGs  resulted  when 

several  agencies  of  the  same  armed  force  contributed  significantly  to 

2 

one  final  product  [X  p  <  .016].  It  appears  that  the  not-for-profit 
firms  are  occasionally  utilized  to  back  up  in-house,  armed  forces  con¬ 
struction  activities.  Given  the  scope,  magnitude,  and  cost  of  many  of 
these  MSGs,  the  extent  of  multiple  authorship  is  less  than  we  had  ex¬ 
pected.  In  fact,  it  may  be  a  manifestation  of  what  appears  to  be  a 
harmful  compar tmentalization  and  overspecialization  in  activity. 

When  the  military  service  responsible  for  an  MSG  is  tabulated 
against  a  range  of  construction-related  descriptive  variables,  the  fol¬ 
lowing  patterns  emerge: 

Construction  and  purpose:  The  Army  is  rather  evenly  split  between 
the  three  purposes  of  technical  evaluation,  doctrine  evaluation,  and 
force-structure  analysis;  the  Air  Force  is  doing  more  force-structure 
analysis,  50.0  percent,  and  technical  evaluation,  42.3  percent,  than 
doctrine  evaluation,  7.7  percent;  and  the  Navy  is  largely  constructing 
technical  evaluations,  77.8  percent,  to  the  exclusion  of  other  types 

[x2  p  <  .0001]. 

Initiation:  Navy  MSGs  are  initiated  either  by  external  users  or 
by  the  builders  themselves  to  a  far  greater  extent  than  in  the  other 
services.  The  Army  accounts  for  11  of  the  17  or  64.7  percent  of  the 
external,  non-user  initiations.  No  particular  pattern  is  discernible 
for  the  Air  Force;  that  is,  initiation  comes  from  a  variety  of  sources 
[X2  p  <  .03].  There  is  also  no  observable  pattern  in  Air  Force  speci¬ 
fication  of  the  purposes  of  its  MSGs.  The  Army,  on  the  other  hand,  ac¬ 
counts  for  two-thirds  of  those  that  are  tightly  specified.  The  bulk  of 

2 

Navy  work  was  noted  as  being  "moderately"  specified  IX  p  <  .002]. 
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Construction  site:  While  not  significantly  related  [X  p  <  .15, 
n.  s .  ] ,  on  a  percentage  basis  the  Air  Force  does  more  of  its  own  work 
in-house  than  do  the  other  services;  the  Army  uses  for-profits  more 
than  expected  and  more  than  the  others;  and  the  Navy’s  involvement  with 
Johns  Hopkins  University  makes  it  the  dominant  university  user. 

Development :  As  noted,  about  two- thirds  of  the  sampled  MSGs  were 
developed  in  18  months  or  less.  The  Army  accounted  for  21  of  the  36 
MSGs  built  in  6  months  or  less.  Navy  construction  apparently  normally 
takes  from  6  months  to  a  year,  while  no  distinct  pattern  is  evident  for 
Air  Force  construction.  Likewise,  at  the  other  end  of  the  scale,  no 
one  service  stands  out  as  taking  particularly  long  to  develop  its  MSGs 
tx2  p  <  .05]. 

Data  sources  and  validity:  Navy  data,  as  indicated  by  the  respon¬ 
dents,  are  most  likely  to  have  been  cross-checked,  field-tested,  or  ex¬ 
perimentally  derived,  27/36  or  75.5  percent.  For  the  other  services, 
it  was  about  a  toss-up  as  to  whether  data  were  checked  or  not.  MSGs 
built  at  or  for  the  Joint  Chiefs  of  Staff  were  about  three  times  as 
likely  not  to  have  their  data  checked  as  were  all  others  in  the  sample, 
in  fact,  in  10  of  11  "other  DOD"  MSGs  the  data  were  not  cross-checked, 
were  unknown,  or  no  response  was  given.  Validation  procedures  were  not 
significantly  service-specific,  although  no  less  than  38  or  28.8  percent 
of  the  entire  sample  reported  that  validation  was  not  undertaken,  was 
unknown,  or  gave  no  response. 

Data  type:  We  asked  what  predominant  type  of  data  were  used, 

whether  hard,  moderate,  soft,  or  combinations.  Examples  of  each  of 

three  basic  types  were  provided.  Only  8  used  mainly  soft  data  or  soft 

data  in  combination  with  numbers  of  greater  certainty,  a  point  already 

noted.  Generally,  Air  Force  MSGs  used  hard  data  to  a  greater  extent 

2 

than  did  those  of  the  Navy  or  Army  [X  p  <  .01]. 

Construction  and  security  classification:  If  an  MSG  without  data 
inputs  was  classified  at  all,  and  about  one-fourth  were,  it  was  most 
likely  at  the  Confidential  (17/37  or  46.0  percent)  or  For  Official  Use 
Only  (9/37  or  24.3  percent)  level.  More  Army  MSGs  are  unclassified 
(81.4  percent)  than  Air  Force  (69.2  percent)  or  Navy  (64.0  percent) 

[X2  p  <  .01],  When  data  inputs  are  added,  a  full  85.6  percent  of  the 
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entire  sample  becomes  classified,  mostly  at  the  Secret  level  (78  or 

59.1  percent).  On  a  percentage  basis,  the  Air  Force  at  92.3  percent 

has  more  classified  MSGs  than  expected  on  the  average  for  the  entire 

2 

sample,  and  more  than  either  the  Navy  or  the  Army  [X  p  <  .002]. 

Size  of  MSG:  The  bimodal  distribution  noted  for  the  whole  sample 

held  for  the  individual  services  with  only  minor  variations;  the  Navy 

at  33  percent,  on  a  straight  percentage  basis,  had  more  MSGs  in  the 

largest  size  category  (>  10,000  instructions)  than  did  either  the  Army 

(28.8  percent)  or  the  Air  Force  (19.2  percent).  Most  Air  Force  entries 

were  in  the  1,000-2,000  instruction  range,  9/26  or  34.6  percent.  If 

Army  MSGs  were  not  large,  and  17/59  were  not,  they  were  in  either  the 

2,000-4,000  or  4,000-6,000  range  (both  8/59  or  13.6  percent)  [X  p  <  .001], 

Levels  of  MSG  resolution:  The  Navy  is  building  more  of  its  MSGs 

at  a  finer  level  of  temporal  detail  than  are  the  other  services;  all  are 

cast  either  in  seconds  (22/36  or  61.1  percent)  or  minutes  (8/36  or  22.2 

2 

percent),  with  the  remaining  6  not  answering  [X  p  <  .001].  The  Army 

builds  most  of  its  MSGs  at  the  level  of  spatial  detail  of  the  kilometer, 

the  Navy  at  the  meter  level,  and  the  Air  Force  shows  no  clear  preference. 

In  keeping  with  the  fine  grain  of  much  of  the  sample,  most  MSGs  were 

cast  at  the  engagement  level.  Battles,  campaigns,  and  wars,  increasingly 

large  levels  of  resolution  for  military  activity,  are  mainly  the  province 

of  the  Army,  which  accounts  respectively  for  85.7  percent,  40.0  percent, 

2 

and  61.1  percent  of  the  activity  in  each  [X  p  <  .001]. 

When  MSG  category  was  tabulated  against  the  ratio  of  model  time  to 
real  time,  a  technical  concept  measuring  the  extent  of  time  compression 
or  expansion,  a  full  30.3  percent  of  the  respondents  either  did  not  know 
or  failed  to  answer.  Of  the  remainder,  if  an  MSG  compressed  time  ex¬ 
tremely,  it  was  probably  a  simulation;  real-time  representations  were 

fairly  evenly  divided  among  the  categories;  and  expanded  MSGs  were  man- 

2 

machine  exercises  or  models  [X  p  <  . 03] . 

The  smallest  MSGs,  those  having  fewer  than  1,000  computer  instruc¬ 
tions,  were  models  about  73  percent  of  the  time.  More  than  half  of  the 
man-machine  exercises  were  in  the  largest  size  category,  54.5  percent, 
while  only  33  percent  of  the  simulations  and  21  percent  of  the  models 
were  as  large  [X2  p  <  .02].  We  looked  at  the  incidence  of  intangible 


-30- 


assumptions  and  MSG  size  and  found  that  for  the  largest  category,  in¬ 
tangible  assumptions  were  made  nearly  20.0  percent  more  than  would  be 

2 

expected  for  the  whole  sample  [X  p  <  .001]. 

OPERATIONS 

Questions  related  to  MSG  operations  dealt  with  two  broad  matters: 
scientific  standards  and  technical-procedural  issues. 

Concepts  and  Questions 

Of  major  concern  was  what  testing  and  professional  controls  were 
employed  to  insure  MSG  fidelity.  Questions  such  as  whether  sensitivity 
testing  was  carried  out,  whether  the  operations  of  a  given  MSG  could  be 
transferred  to  a  comparable  location  elsewhere,  and  whether  an  indepen¬ 
dent  professional  review  had  been  carried  out  are  examples  of  our  opera¬ 
tional  control  questions.  Questions  about  the  MSG!s  need  for  special 
facilities,  languages,  or  documentation  pertained  more  to  techniques 
and  hardware. 

Results 

Description.  Sensitivity  analysis  is  an  important  operational 
control,  especially  when  the  number  of  variables  is  large  and  the  model 
complex.  One  must  find  out  about  an  MSG’s  behavior  as  sets  of  input 
parameters  are  altered  in  interesting  ways.  Without  sensitivity  analy¬ 
sis,  one  cannot  know  much  about  the  MSGfs  performance.  In  our  sample, 

45  percent  indicated  that  their  MSG  had  not  been  sensitivity-tested. 
Granted,  such  testing  is  generally  expensive  and  time-consuming,  but 
it  is  essential  to  a  determination  of  the  MSGfs  usefulness  and  validity. 

We  have  already  touched  on  the  topic  of  data  validity.  Probing 
a  bit  more,  we  found  that  about  14  percent  used  several  generally  ac¬ 
ceptable  and  commendable  means  of  checking  on  the  accuracy  and  quality 
of  input  data.  In  about  28  percent  of  the  cases,  less  thorough  measures 
were  taken,  because  questions  remained  about  the  precision  of  some  num¬ 
bers  actually  being  used.  For  30  percent,  the  written  commentary  com¬ 
bined  with  categorical  assessments  indicated  that  the  issue  was  at  least 
considered  and  some  effort  made  to  check  out  the  data.  Data  validation 
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was  not  attempted  in  16  percent  of  the  cases,  and  a  full  13  percent 
either  did  not  reply  or  did  not  know  about  the  matter. 

Documentation  is  a  prime  control  function  that  has  received  in¬ 
adequate  attention.  Figure  8  gives  a  clear  portrayal  of  what  appears 
to  be  the  optimistically  biased  self-assessments  of  most  respondents. 

These  assessments  would  be  more  believable  if  they  had  been  put  to, 
and  had  passed,  an  operational  test.  In  fact,  as  shown  in  Fig.  9,  only 
18.2  percent  of  the  MSGs  were  generally  transferable.  The  importance 
of  replication  is  both  scientific  and  administrative.  If  a  model  can¬ 
not  be  transferred,  independent  reviewing  is  more-or-less  precluded. 

The  administrative  issue  is  the  obvious  one  of  controlling  redundancy; 
some  52  percent  of  the  respondents  indicated  that  they  were  not  aware 
of  any  closely  related  MSG,  and  approximately  4  percent  did  not  reply 
at  all.  But  that  is  of  little  consequence  because  nearly  half  the  re¬ 
spondents,  as  depicted  in  Fig.  10,  did  not  know  what  it  would  cost  to 
transfer  operations. 

Responses  indicate  that  nearly  half  the  MSGs  had  not  been  sub¬ 
jected  to  review  outside  the  building-user  organization.  The  actual 
figure  is  probably  higher,  since  this  sample  is  biased  toward  Army 
models,  which  probably  have  benefited  from  the  recent  ad  hoc  Army 
Model  Review  Committee Ts  efforts  to  increase  the  outside  professional 
review  of  MSGs.  At  best  there  is  less  than  a  50/50  chance  that  any 
active  model  in  the  inventory  has  been  reviewed.  The  date  of  most 

recent  review  is  suggested  in  Fig.  11. 

Special  facilities  of  one  variety  or  another  were  required  for  about 
a  third  of  the  sample.  About  16.6  percent  needed  dedicated  computational 
systems,  13.0  percent  used  specialized  languages  or  libraries,  and  2.3 
percent  required  special  buildings  or  laboratory  facilities.  MSGs  should 
be  made  more  transferable  than  they  are:  there  are  few  valid  technical 
obstacles  to  replication. 

Trends .  There  has  been  a  significant  decline  since  1966  in  the 
incidence  of  external  professional  review,  in  spite  of  the  recent  ad  hoc 
Army  initiatives.  Percentage  data  are  summarized  in  abbreviated  form 
in  Table  1. 

* Chaired  by  Dr.  J.  Honig  at  the  request  of  Lieutenant  General 
William  E.  DePuy,  U.S.  Army. 
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pjg  t  8 — Quality  of  MSG  documentation:  Respondents'  assessments 

(variable  DOCEXT) 


Fig .  9—  Transferability  of  MSG:  Respondents'  assessments  (variable  TRANSU) 
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Fig.  10 — Estimated  transfer  costs  (production  cost  plus  increment) 

(variable  TRANSC) 


Fig.  11 — Time  (months)  since  last  external  professional  review 

(variable  REVDAT) 
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Table  1 

TRENDS  IN  EXTERNAL  PROFESSIONAL 
REVIEWING  OF  MSGs 


Incidence 

1966-1967 

(%) 

1971-1972 

(%) 

A  % 

No  review 

22.7 

50.0 

+27.3 

Review 

59.1 

38.9 

-20.2 

No  response 

18.2 

11.1 

-7.1 

Since  1966,  there  has  been  a  slight  decrease  in  the  percentage  of 
MSGs  not  intended  for  transfer  and  those  that  are  generally  transferable. 
It  appears  that  the  extremes  are  moving  toward  a  middle  category  of  dif¬ 
ficulty  of  transfer. 

The  lack  of  knowledge  about  transfer  costs  has  remained  steady  at 
about  40  percent  in  the  same  period.  In  other  words,  a  consistently 
high  proportion  of  respondents  had  no  idea  what  it  would  cost  to  repli¬ 
cate  their  MSG  elsewhere. 

The  descriptive  attributes  for  data  sources,  data— validation  pro¬ 
cedures,  extent  of  documentation,  and  incidence  of  sensitivity -testing 
remained  stable  over  the  period  of  the  analysis. 

Very  little  has  ever  been  published  in  journals  or  books  about  the 
MSGs  in  this  sample.  However,  since  1966,  there  has  been  a  major  per¬ 
centage  increase  in  use  of  the  Defense  Documentation  Center.  The  pri¬ 
mary  form  of  documentation  has  been  relatively  stable  over  time——  re¬ 
ports  only"  account  for  25  to  30  percent,  "user  manuals  only,"  slightly 
less  than  10  percent~but  "user  manual  plus  program  decks"  as  a  category 
increased  from  13  to  28  percent  in  the  period. 

Relationship  to  Other  Descriptors.  Here  we  consider  the  linkage 
of  MSG  operational  characteristics  and  the  military  service  responsible 
for  the  MSG,  and  the  effects  of  external  professional  reviewing  on  the 
sampled  population. 

No  significant  relationships  were  observed  between  the  responsible 
service  and  any  of  the  following  operations-oriented  descriptors:  ease 
of  transfer,  cost  of  transfer,  incidence  of  sensitivity  testing,  and 
specialized  facilities. 
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The  amount  of  external  professional  reviewing  initiated  by  each 
service  was  significant.  Based  on  the  average  for  the  entire  sample 
(see  Table  2) ,  Air  Force  MSGs  are  least  likely  to  have  benefited  from 
external  reviewing.  Adding  the  number  of  no-responses  to  the  number 
of  MSGs  not  reviewed  reveals  that  over  80  percent  of  the  sampled  Air 
Force  activity  was  evidently  not  subjected  to  external  scrutiny.  The 
data  also  indicate  both  the  extent  of  the  Army  Models  Review  Committee's 
recent  efforts  and  the  low  rate  of  review  for  other  DOD  MSGs. 

Table  2 

EXTERNAL  PROFESSIONAL  REVIEWING,  BY  SERVICE 


Service 

MSGs 

Not 

Reviewed 

MSGs 

Reviewed 

No 

Response 

Total 

USA 

15 

36 

8 

59 

USAF 

18 

5 

3 

26 

USN 

13 

21 

2 

36 

Other  DOD 

7 

3 

1 

11 

Total 

53 

65 

14 

132 

[X2  p  <  .003] 


When  service  is  tabulated  against  the  date  of  latest  professional 
review,  we  see  that  25.4  percent  of  the  Army's  entries  were  reviewed 
within  the  six  months  prior  to  the  cutoff  date  of  this  survey  around 
October  1971 — a  significantly  higher  percentage  than  those  of  either 
the  Navy  or  Air  Force. 

Of  the  5  Air  Force  entries  that  were  given  external  professional 
review,  3  were  last  looked  at  between  one  and  two  years  ago,  1  more 
than  three  years  ago,  and  1  less  than  six  months  prior  to  the  survey  s 
cutoff  date.  The  portion  of  Navy  MSGs  reviewed  is  consistent  over  time 

[X2  p  <  .05]. 

The  incidence  of  external  professional  review  relates  to  other 

characteristics  of  the  sample  as  well. 

While  MSGs  used  for  analysis  and  diagnosis,  teaching  and  training, 
or  operations  stood  about  a  50/50  chance  of  being  reviewed,  those  with 
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research  applications  did  not  fare  so  well.  Of  the  7  research  or  ex¬ 
perimentation  MSGs,  only  1  had  had  a  professional  review.  It  is  curi¬ 
ous  indeed  that  the  few  explicitly  scientific  examples  were  not  sub¬ 
jected  to  this  rudimentary  scientific  control  procedure. 

Professional  reviewing  was  not  significantly  related  to  transfer- 
ability,  transfer  cost,  size  of  the  MSG,  total  cost  to  build,  extent 
of  documentation,  type  of  data  mainly  used,  or  the  incidence  of  sensi¬ 
tivity  testing. 

There  is  a  significant  relationship  between  the  data  sources  of 
an  MSG  and  its  later  receiving  external  review.  If  an  MSG  (1)  under¬ 
went  checking  of  data,  (2)  derived  from  field  exercises,  or  (3)  used 
data  from  a  variety  of  experimental  and  operational  sources,  the  chances 
were  doubled  that  a  professional  review  would  be  done  [X  p  <  .005]. 
Likewise,  if  data  validation  procedures,  as  described  in  written  com¬ 
mentary,  were  highly  or  moderately  rigorous  (and  55  or  41.6  percent 
were  so  classed) ,  the  chances  of  subsequent  professional  review  were 
also  nearly  double  those  of  MSGs  having  less  rigorously  validated  data 
[X2  p  <  .02],  This  suggests  that  concern  for  rigorous  design  and  pro¬ 
duction  carries  over  into  operational  control  procedures  as  well. 

Unfortunately,  that  suggestion  does  not  hold  for  sensitivity  test¬ 
ing,  which  was  not  clearly  related  to  professional  reviewing.  Curiously, 
sensitivity  testing  was  strongly  related  to  MSG  size.  MSGs  having  less 
than  4,000  instructions,  i.e.,  the  relatively  smaller  entries,  were 
about  twice  as  likely  not  to  have  had  some  sensitivity  testing  as  the 
larger  MSGs  having  4,000-10,000  instructions.  The  largest  size  cate¬ 
gory  (greater  than  10,000  instructions)  was  evenly  split  on  the  ques¬ 
tion  [X2  p  <  .001].  It  is  not  readily  explicable  that  the  procedure 
is  carried  out  less  frequently  than  expected  for  the  smaller  and  prob¬ 
ably  more  readily  and  inexpensively  tested  MSGs. 

Documentation  and  the  incidence  of  professional  reviewing  were 

related  in  several  interesting  ways. 

MSGs  that  had  been  reviewed  were  twice  as  likely  as  those  not  re¬ 
viewed  to  have  their  documentation  located  in  the  Defense  Documentation 
Center,  and  65  MSGs  in  all  received  some  sort  of  external  professional 
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review.  When,  for  this  subset,  assessments  of  data-validation  proce¬ 
dures  are  tabulated  against  the  location  of  documentation,  a  highly 
significant  finding  emerges.  Of  the  34  MSGs  thought  to  have  high-  and 
moderate— quality  data  validation,  no  less  than  19  have  documentation 
that  is  not  generally  accessible,  being  either  proprietary /not-for- 
profit  organization  (3) ,  proprietary/contact  author  (3) ,  proprietary 
(classified) /contact  author  (10),  unknown/not  available  (2),  or  out-of 
print  (1)  [X2  p  <  .001].  Once  more  we  begin  to  take  a  measure  of  the 
extent  of  isolation  of  much  current  activity.  Of  the  35  MSGs  whose 
documentation  was  public,  located  in  the  Defense  Documentation  Center, 
only  15  were  credited  with  high  or  moderate  data— validation  procedures 
significantly  fewer  than  those  having  inaccessible  or  limited-access 
documentation.  This  finding  is  interesting  enough  to  warrant  repro¬ 
ducing  Table  3  in  full. 


Table  3 

CROSS-TABULATION  OF  DOCUMENTATION  LOCATION  AND  QUALITY  OF  DATA  VALIDATION 
FOR  MSGs  THAT  WERE  EXTERNALLY  REVIEWED 

(N  -  65) 


Quality 

of  Data  Validation  (Variable  DATAV) 

Documentation 

Location 

(Variable  D0CL0C1) 

Unknown 

High 

Moderate 

Weak 

Not 

Done 

No  Re¬ 
sponse 

Total  (%) 

Unknown 

1 

1 

1 

2 

1 

0 

6 

(9.2) 

Out  of  print 

0 

1 

0 

2 

0 

0 

3 

(4.6) 

Proprietary;  not- 
for-profit  or¬ 
ganization 

0 

0 

3 

0 

0 

0 

3 

(4.6) 

Proprietary;  con¬ 
tact  author 

2 

1 

2 

0 

0  1 

6 

(9.2) 

Proprietary 
(classified) ; 
contact  author 

0 

4 

6 

1 

0 

0 

11 

(16.9) 

Public;  in  DDC 

2 

3 

12 

8 

9 

1 

35 

(53.8) 

No  response 

0 

0 

0 

0 

0 

1 

1 

(1.5) 

Total 

5 

10 

24 

13 

11 

2 

65 

(2) 

(7.7) 

(15. A) 

(36.9) 

(20.0) 

(16.9) 

(3.1) 

(100.0) 

NOTE:  Raw  chi  square  -  61.83,  with  30  deg  freedom.  Significance  -  0.0005. 
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USE 

The  question  of  use  is  a  difficult  one.  There  is  little  consensus 
on  answers  to  many  of  the  questions  we  raised  and  little  evidence  that 
they  are  even  being  discussed.  Our  intention  in  this  section  and  the 
one  on  cost  that  follows  is  primarily  to  initiate  discussion  on  these 
neglected  matters,  and  secondarily  to  take  a  reading  on  what  appears 
to  be  the  present  state  of  affairs. 

Concepts  and  Questions 

Besides  asking  whether  an  MSGfs  use  corresponded  to  the  major  pur¬ 
pose  for  which  it  was  constructed,  we  created  other  measures  of  use. 

One  standard  is  the  market  measure — Will  the  MSG  sell;  is  its  fund¬ 
ing  regularly  renewed?  Not  exactly  an  ideal  criterion,  it  provides  a 
crude  pragmatic  indication  of  the  client’s  satisfaction  and  willingness 
to  buy  and  use  more  of  the  same. 

Two  operational  questions  approached  the  matter  slightly  differ¬ 
ently.  The  question,  "How  many  briefings  were  given,  based  on  the  re¬ 
sults  produced  by  the  MSG?"  may  give  one  an  idea  of  how  much  stock  a 
user  places  in  a  particular  MSG.  If  no  briefings  resulted,  that  tells 
something  about  how  key  participants  assess  a  given  enterprise.  The 
corollary  question,  "How  many  times  is  the  MSG  referred  to  in  making 
specific  operational  decisions?"  further  refines  and  clarifies  this 
measure.  To  push  it  to  something  like  an  ideal  criterion,  one  should 
interview  all  relevant  participants  to  determine  precisely  how  an  MSG 
related  to  a  specific  decision,  who  advocated  it,  who  voiced  reserva¬ 
tions,  and  what  official  rationalizations  derived  from  it.  We  had 
neither  the  time  nor  other  resources  to  do  that.  The  questions  of  who 
initiated  the  MSG  and  for  what  purpose  come  to  bear  at  this  point  to 
dramatize  an  MSG's  use.  Straightforward  questions  such  as,  "Is  the 
MSG  active  or  not?"  and  "How  often  is  it  operated?"  provide  further 
important  information. 

Operations  and  use  overlap  when  considering  what  kinds  of  documen¬ 
tation  are  located  where,  how  easy  it  is  to  transfer  use  from  one  site 
to  another,  and  the  rate  at  which  and  reasons  why  an  MSG  becomes  obso¬ 


lescent. 
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Results 

Description.  Primary  categories  of  actual  use  are  shown  in  Fig. 
12.  The  emphasis  among  initiators  on  the  use  of  MSGs  for  analysis  and 
diagnosis  is  reflected  nearly  exactly  in  the  way  respondents  indicated 
MSGs  are  being  used.  The  other  possible  primary  utilization  categories 
taken  together  account  for  less  than  22.0  percent  of  the  sample.  Re¬ 
spondents  indicated  that  42  or  31.8  percent  of  the  MSGs  have  secondary 
uses  and  that  13  or  9.8  percent  have  tertiary  uses.  Of  those  with  sec¬ 
ondary  uses,  30  are  used  for  operations,  5  for  analysis  and  diagnosis, 

4  for  research  and  theory  development,  and  3  for  experimentation. 


Fig.  12 — Respondents1  primary  classification  of  MSG  use 
(variable  USE  1 ) 


About  80  percent  of  the  respondents  were  highly  or  absolutely 
certain  about  how  their  MSGs  are  being  used. 

Queries  about  genealogy  (discussed  on  p.  22,  above)  revealed  that 
nearly  three-fourths  of  the  sample  have  a  direct  antecedent  and  about 
one-third  have  already  spawned  offspring.  The  crude  market  measure 
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of  utilization  tells  us  that  business  is  good  and  clients  are  suffi¬ 
ciently  contented  to  keep  funds  flowing. 

Several  questions  were  asked  about  briefings.  For  some  models 
the  number  of  briefings  may  not  be  a  good  measure,  but  for  games  and 
simulations  it  usually  is.  Frequently  a  briefing  may  be  no  more  than 
a  superficial  performance,  but  at  least  it  indicates  that  someone  con¬ 
sidered  the  work  interesting  enough  to  produce.  To  the  simplest  ques¬ 
tion  about  the  total  number  of  briefings  based  on  the  particular  MSG 
(see  Fig.  13),  11.4  percent  of  the  respondents  replied  "none,"  and 
42.4  percent  simply  did  not  know. 


Number  of  respondents 


Fig.  13 — Total  number  of  briefings  based  on  MSGs 

(variable  BRIEF)  _ 

Respondents  were  asked  for  written  commentary  on  the  level  (or¬ 
ganization  and  personnel)  and  purpose  (decisions  dependent  on  MSG)  of 
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the  briefings  that  were  given.  The  61  MSGs  for  which  answers  were 
given  were  commonly  briefed  to  the  project  staff,  to  an  immediate  su¬ 
perior  in  the  organizational  hierarchy,  or  occasionally  to  flag-level 
officers  and  their  civilian  counterparts  at  the  top  command  levels  of 
the  services.  The  purposes  of  those  briefings  were  not  reported  con¬ 
sistently  enough  to  allow  simple  coding  and  categorization,  so  verbal 
summary  abstracts  were  recorded  for  each  MSG  by  name.  Examples  of  com¬ 
mon  replies  include  the  following:  used  for  input  to  other  studies 
(SEANITEOPS) ,  strategic  posture  choices  (NEWCON) ,  information  on  F  15 
performance  (TAC  AVENGER) ,  no  decisions  based  on  model  (AREA  DOMINATION 
II) ,  and  selection  of  weapons  for  DLGN-25  (AAWSEM) . 

Generally,  briefings  appear  to  have  been  generated  for  the  benefit 
of  one  or  a  few  colonels  or  generals  (or  their  naval  and  civilian 
counterparts)  who  occupy  studies  and  analysis  roles  in  their  respective 
services.  What  further  use  these  audiences  made  of  the  briefing  infor¬ 
mation  is  not  determinable  from  the  questionnaire  answers  and  would  re¬ 
quire  extensive  personal  interviewing  to  resolve. 

Let  us  reiterate  that  nearly  half  the  MSGs  surveyed  had  not  pro- 

duced  a  single  briefing. 

The  question  of  who  initiates  the  development  of  an  MSG  was  dis¬ 
cussed  earlier  (p.  17).  As  revealed  in  Fig.  3,  analytic  and  diagnostic 
purposes  predominate.  Comparing  those  data  with  responses  m  Fig.  12 
suggests  that  to  a  marked  degree  initiators  are  getting  what  they  re- 

quested. 

The  frequency  of  MSG  operation  suggests  another  dimension  of  use. 

A  distinctly  bimodal  distribution  is  evident  in  the  data  shown  in  Fig. 

14  on  the  average  annual  frequency  of  MSG  use. 

Limitations  on  the  availability  of  documentation  and  problems  of 
transferability,  touched  on  when  describing  operations,  do  little  to 
promote  widespread  use.  We  asked  about  the  availability  and  location 
of  documentation.  Public  availability  was  concentrated  in  the  Defense 
Documentation  Center;  no  MSGs  had  documentation  in  the  Library  of 
Congress;  and  only  one  was  listed  as  having  been  written  up  in  journals 
or  books.  Multiple  sources  of  documentation  were  listed,  but  only  three 
respondents  checked  two  sources,  and  only  two  checked  three  separate 


sources • 
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Number  of  respondent's 


Fig  .  14 — Average  frequency  of  MSG  operation  per  year  (variable  OPFRQ1) 

The  cost  of  keeping  an  MSG  up-to-date  and,  to  a  lesser  extent,  the 
average  cost  of  running  an  MSG  bear  on  its  utilization.  To  get  at  an¬ 
nual  update  costs,  we  asked  for  information  on  total  dollars,  profes¬ 
sional  man-years,  and  programmer  man-years  expended  over  and  above 
"normal”  operating  costs.  For  the  71  MSGs  for  which  replies  were  given, 
cost  data  are  summarized  in  Fig.  15.  It  is  clear  that  updating  is  not 
a  trivial  matter;  likewise,  the  sampled  MSGs  are  generally  expensive  to 
run.  Figures  on  the  cost  of  a  single  MSG  run,  assuming  only  minor  or 
no  variations  in  input  values,  are  shown  in  Fig.  16. 

Trends.  The  availability  and  location  of  documentation  have  changed 
somewhat  for  MSGs  that  have  become  operational  since  1966.  On  a  per¬ 
centage  basis,  more  documentation  is  being  located  in  the  Defense 


-43- 


Fig.  15 — Total  annual  MSG  update  costs  (in  $  thousands) 
(variable  CSTUP) 
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Fig.  16— Cost  of  a  single  run  of  MSG  (in  $  thousands) 
(variable  CSTRUN) 
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Documentation  Center,  but  more  also  is  proprietary,  classified,  and 
available  only  from  the  author.  Fewer  MSGs  have  documentation  that 
is  proprietary,  available  from  a  not-for-profit  firm  and  proprietary, 
or  unclassified  and  available  only  from  the  author. 

Average  annual  update  costs  are  changing  in  significant  ways. 
Fewer  MSGs,  on  a  percentage  basis,  are  updated  for  less  than  $10,000 
now  than  in  1966  (16.7  percent  now  versus  36.4  percent  then),  but  no 
MSGs  currently  fall  in  the  $50, 000-$100,000  range.  What  has  happened 
is  that  MSGs  in  the  two  intermediate  ranges  have  increased,  as  shown 
in  Table  4.  Other  trends  in  use  are  not  so  clearly  discernible. 


Table  4 

TRENDS  IN  THE  COST  OF  ANNUAL  UPDATE 


Cost 

($  thousands) 

1966-1967 

(%) 

1971-1972 

(%) 

4% 

$  <  10 

36.4 

16.7 

-19.7 

$10- $24 

9.1 

11.1 

+2.0 

$25-$49 

9.1 

22.2 

+13.1 

Relationship  to  Other  Descriptors.  Respondents  were  highly  or 
absolutely  certain  about  the  actual  use  of  an  MSG  in  nearly  80  percent 
of  the  cases.  No  significant  association  was  noted  between  category 
of  use  and  level  of  confidence. 

The  correspondence  between  the  initiator’s  intended  purpose  and 
the  actual  use  to  which  the  MSG  was  put  was  strikingly  consistent.  If 
someone  wanted  an  MSG  for  analysis  and  diagnosis,  he  got  it  97/108  or 
89.8  percent  of  the  time.  The  other  11  MSGs  ended  up  being  used  pri¬ 
marily  for  teaching  and  training  (3),  operations  (3),  experimentation 
(1) ,  research  and  theory  development  (1),  or,  the  respondent  did  not 
know  or  did  not  reply  (3)  [X2  p  <  .001].  Those  deflections  of  intent 
are  minor  indeed. 

When  the  respondent’s  role  was  tabulated  against  his  assessment  of 
the  quality  of  his  MSG’s  documentation,  we  found  that  funders  and  spon¬ 
sors  were  the  most  likely  to  claim  excellent  documentation  rather  than 
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any  other  possible  quality  category,  7/19  or  36,8  percent.  Users  in¬ 
dicated  that  documentation  was  very  good  more  frequently  than  other 
categories,  16/42  or  38.1  percent.  Designer /builders  opted  for  aver¬ 
age,  14/41  or  34.1  percent,  and  very  good,  12/41  or  29.3  percent,  in 
nearly  equal  measure.  However,  caretakers  indicated  predominantly  that 
documentation  was  either  average,  8/18  or  44.4  percent,  or  poor,  4/18 
or  22.2  percent  [X2  p  <  .001].  While  several  interpretations  are 
plausible,  it  seems  that  those  farthest  removed  from  actual  MSG  use 
(funder— sponsor)  are  the  most  glowing  in  their  assessments  of  documen¬ 
tation  quality. 

On  the  average,  about  one- third  of  the  respondents  did  not  know 
whether  their  MSG  had  been  briefed  or  not.  When  those  for  which  no 
briefings  had  been  given  were  totaled  with  this  unknown  category  and 
then  tabulated  against  responsible  military  service,  the  information 
in  Table  5  was  produced.  The  relationship  is  not  statistically  sig¬ 
nificant,  but  it  suggests  that  the  MSGs  in  this  sample  are  either  not 
being  used  or,  more  likely,  that  people  responsible  for  their  use  are 
not  keeping  track  of  how  often  they  are  used. 

Table  5 

BRIEFINGS  BASED  ON  MSGs,  BY  SERVICE 


Service 

MSGs 

Not  Briefed 
or  Unknown 

MSGs 

Briefed 

Total 

USA 

34 

25 

59 

USAF 

12 

14 

26 

USN 

14 

22 

36 

Other  DOD 

11 

0 

11 

Total 

71 

61 

132 

Annual  frequency  of  operation  and  update  costs  are  related  to  the 
individual  services  in  significant  and  interesting  ways.  The  bimodal 
nature  of  annual  use  holds  for  all  services.  A  given  MSG  will,  about 
25  percent  of  the  time,  be  operated  one  to  five  times  a  year,  if  at 
all;  at  the  other  extreme,  it  has  about  a  15  percent  chance  of  being 
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operated  in  excess  of  one  hundred  times  a  year  [X  p  <  .05].  Annual 
updating  costs  are  approximately  the  same  for  all  services.  For  their 
part  of  the  total  sample,  30.5  percent  of  the  Army’s  MSGs  cost  less  than 
$10,000,  compared  with  34.6  percent  of  the  Air  Force’s  and  22.2  percent 
of  the  Navy’s.  The  next  most  frequent  cost  range  is  $25,000-$49,000, 
which  accounts  for  about  12  percent  of  each  service’s  MSGs  [X  P  <  .005]. 

COSTS 

A  somewhat  cynical,  but  wise,  gamester  of  our  acquaintance  claims 
that  gaming  activities  cost  whatever  the  bookkeepers  want  to  make  them 
cost.  There  are  many  lengthy,  complicated  reasons  validating  that  re¬ 
mark,  but  suffice  it  to  say  that  costs  are  deceptively  hard  to  deter¬ 
mine. 

As  the  work  involves  many  invisible  costs,  overhead  costs,  jointly 
shared  facilities,  and  jointly  used  products,  formulating  a  meaningful 
costing  procedure  poses  deep  scientific  problems  that  are  far  from 
being  resolved.  The  respondents  to  our  survey  found  it  difficult  even 
to  give  a  simple  cost  description. 

Concepts  and  Questions 

The  questionnaire  contained  a  large  number  of  cost  questions.  Be¬ 
sides  attempting  to  characterize  the  nature  of  costs  using  several 
plausible  dimensions  and  categories,  we  were  interested  in  determining 
the  amount  of  current  attention  to  costs  and  the  general  level  of 
knowledge  in  the  profession  about  costs. 

Answers  on  all  three  topics— nature,  attention,  and  knowledge— 
were  disappointing.  More  than  a  third  of  the  respondents  simply  did 
not  answer  cost  questions,  and  of  those  who  did,  the  variability  of 
replies  was  large  and  the  level  of  confidence  was  low.  If  these  qual¬ 
ified  professionals,  being  asked  cost  questions  by  the  U.S.  General 
Accounting  Office  on  behalf  of  the  House  Appropriations  Committee, 
could  not  come  up  with  some  sort  of  figures,  it  is  difficult  to  see 
who  can,  unless  cost  records  are  kept  with  the  work. 
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Results 

Description.  Though  nearly  three-fourths  of  the  MSG  sample  had 
at  least  one  direct  antecedent,  62.1  percent  of  the  respondents  either 
did  not  know  ar  failed  to  respond  to  a  question  about  the  costs  for 
these  MSG  families.  Moreover,  of  those  that  answered,  uncertainty 
about  the  answers  was  high;  over  half  had  low  confidence  in  their  re¬ 
sponses.  This  suggests  that  cost  considerations  are  discontinuous, 
i.e. ,  costs  are  not  accumulated,  even  though  changes  in  the  MSG  may  be 
marginal  from  one  version  to  the  next.  Costs  easily  become  separated 
from  substance. 

Direct  funds,  money  formally  assigned  for  construction  purposes, 
were  used  by  61  or  46.2  percent  of  the  sample.  How  these  funds  were 
distributed  is  shown  in  Fig.  17.  Confidence  levels  were  low  in  nearly 
half  the  cases . 

Total  or  gross  costs— direct ,  indirect,  imputed,  and  unimputed— 
are  depicted  in  Fig.  18.  The  confidence  level  for  this  roughest  approx¬ 
imation  of  costs  was  slightly  better  than  for  the  other  more  detailed 
cost  categories  but  considerably  worse  than  for  other  kinds  of  vari¬ 
ables.  For  instance,  on  the  classification  of  MSG  purpose,  80  percent 
were  either  highly  or  absolutely  certain.  On  total  costs,  only  20.4 
percent  were  as  confident. 

Other  cost  categories  have  been  discussed  in  earlier  sections. 

For  example,  man-years  expended  to  construct  an  MSG  and  sources  of  funds 
were  mentioned  under  "Production" ;  transfer  costs  and  special  facilities 
under  "Operations";  and  update  and  single-run  costs  under  "Use.n  What 
those  sections  reveal  only  confirms  our  impression  that  (1)  generally 
accepted,  simple  accounting  definitions  do  not  exist,  (2)  insufficient 
attention  is  paid  to  costs  to  manage  these  MSGs  effectively,  and  (3)  few 
people  know  how  much  money  was  invested  in  their  MSG,  how  much  is  being 
spent  to  operate  and  maintain  it,  and  what  monetary  benefits  are  ac 
cruing.  It  appears  that  elementary  questions  such  as  "What  was  bought 
for  how  much?"  seldom  get  asked. 

Trends.  Trends  in  three  categories  are  worth  considering:  total, 
transfer,  and  update  costs.  The  incidence  of  "unknown"  and  no  responses 
for  total  cost  information  on  all  MSGs  is  steady  at  around  25  percent 
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fig.  17 — Direct  funds  for  MSG  construction  (variable  FUNDIR) 


Fig.  18 — Total  MSG  costs:  Direct,  indirect,  imputed,  unimputed 

(variable  TOTCST) 
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for  the  years  after  1964,  In  other  words,  a  fourth  of  the  respondents 
did  not  even  bother  to  guess#  The  proportion  of  MSGs  in  each  of  the 
cost  ranges  has  remained  relatively  stable  over  time.  Unknowns  and  no¬ 
responses  regarding  transfer  costs  increased  over  those  regarding  total 
costs  to  45  percent  for  MSGs  initiated  after  1966.  The  amount  of  no¬ 
responses  and  unknowns  regarding  update  costs  is  about  steady  at  40 
percent. 

Relationship  to  Other  Descriptors.  Because  so  little  information 
was  given,  fewer  statistical  tables  were  generated  for  cost  than  for 
other  descriptive  variables.  No  significant  associations  existed  between 
total  costs  and  the  incidence  of  professional  reviewing,  data  sources, 
or  general  classification  (model,  simulation,  man-machine  game,  etc.). 

When  military  service  was  tabulated  against  direct  costs,  the  data 

in  Table  6  were  generated.  When  the  process  was  repeated  for  total 

costs  (Table  7),  the  pattern  shifted  somewhat.  Not  only  were  there 

more  responses,  but  the  distribution  changed.  The  information  in  this 

2 

table  is  not  statistically  significant  [X  p  <  .09,  n.s.]  but  neverthe¬ 
less  worth  presenting. 

OPINIONS  OF  RESPONDENTS 

We  asked  the  respondents  their  professional  opinions  about  a  num¬ 
ber  of  issues,  including  the  potential  usefulness  of  clearinghouses, 
regional  centers,  and  external  professional  review  boards.  We  built 
redundancy  into  the  questionnaire  in  this  regard,  asking  similar  ques¬ 
tions  in  different  portions  of  the  questionnaire  in  slightly  different 
ways,  to  provide  consistency  checks. 

Description  of  Results 

On  the  question  of  establishing  a  clearinghouse  to  coordinate  in¬ 
formation  about  all  MSG  activities  within  the  Department  of  Defense, 
respondents  were  generally  quite  favorably  disposed,  as  shown  in  Fig* 

19.  More  than  half  thought  it  would  be  useful  or  highly  useful. 

Twice  we  raised  the  issue  of  standardization.  The  first  time,  it 
was  in  the  question,  "What  is  your  belief  in  the  advisability  of  in¬ 
creasing  DOD  gaming  and  simulation  activity  for  standardization?"  Re¬ 
spondents  were  asked  to  check  one  category  on  a  five-point  scale  from 
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Table  6 

CROSS-TABULATION  OF  MILITARY  SERVICE  AND  DIRECT  COSTS 


Direct  Costs  (Variable  CSTDIR) 


Service 

■ 

$100- 

$249K 

$250- 

$499K 

E 

1223 

$750- 

$999K 

H 

Unknown 

NR 

Total  (Z) 

USA 

12 

12 

— 

n 

5 

0 

i 

18 

59  (44.7) 

USAF 

5 

0 

2 

0 

0 

0 

i 

18 

26  (19.7) 

USN 

8 

10 

9 

u 

2 

2 

i 

3 

36  (27.3) 

Other 

DOD 

1 

■ 

3 

■ 

■ 

■ 

0 

3 

11  (8.3) 

Total 

26 

9 

8 

3 

3 

42 

132 

(2) 

(19.7) 

(6.8) 

(6.1) 

(2.3) 

(2.3) 

(31.8) 

(100.0) 

NOTE:  Raw  chi  square  «  47.42,  with  21  deg  freedom.  Significance  *  0.0008. 


Table  7 

CROSS-TABULATION  OF  MILITARY  SERVICE  AND  TOTAL  COSTS 


Total  Costs  (Variable  TOTCST) 


Service 

<  $49K 

$50- 

$99K 

V' 

IPPP3 

$250- 

$499K 

H 

$750- 

$999K 

$1- 

$2.49 

million 

$2.5- 

$5 

million 

Unknown 

NR 

Total  (Z) 

USA 

15 

6 

7 

■■ 

1 

4 

1 

14 

59  (44.7) 

USAF 

10 

6 

2 

1 

0 

1 

0 

6 

26  (19.7) 

USN 

4 

6 

10 

8 

BB 

2 

2 

0 

3 

36  (27.3) 

Other 

DOD 

2 

■ 

3 

■ 

0 

2 

0 

0 

2 

11  (8.3) 

Total 

31 

19 

22 

B 

5 

5 

7 

1 

25 

132 

(2) 

(23.5) 

(14. 4)1 

(16.7) 

(3.8) 

(3.8) 

(5.3) 

(0.8) 

(18.9) 

(100.0) 

«  33.36,  with  24  deg  freedom.  Significance  *  0.0965. 


NOTE:  Raw  chi  square 
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Number  of  respondents 

pjg #  19— Respondents'  opinions  on  MSG  clearinghouses 
(variable  CLEAR) 

highly  useful  to  highly  harmful.  The  second  time,  near  the  end  of  the 
questionnaire,  the  question  was  phrased,  nIs  it  premature  to  try  to 
form  a  professional  standards  committee  for  models,  games,  and  simu¬ 
lations?  Is  it  needed?  Would  it  probably  do  good  or  harm?"  This 
time  we  asked  for  written  commentary,  which  was  later  coded  as  highly 
useful,  useful,  same  as  present,  harmful,  highly  harmful,  or  no  response. 
The  comments  gave  insight  into  why  there  was  such  pronounced  opposi¬ 
tion  to  standardization,  as  evidenced  in  Fig.  20. 

We  also  asked  for  opinions  on  the  initiation  of  regional  centers 
to  coordinate  production,  operation,  and  use.  Some  60  percent  thought 
such  centers  would  be  harmful  or  highly  harmful.  Another  question 
asked  respondents  whether  the  creation  of  external  reviewing  boards 
would  be  an  improvement.  Some  57  percent  opposed  the  idea,  terming  it 
harmful  or  highly  harmful  and  impossible  to  staff  adequately. 

On  each  question  we  asked  respondents  to  amplify  their  categori¬ 
cal  responses  with  written  commentary.  The  prevailing  attitude  of  those 
opposed  to  clearinghouses  was  that  it  would  add  an  unnecessary  layer  to 
the  existing  bureaucracy.  Roughly,  the  sentiment  was,  "In  theory  it 
sounds  fine,  but  in  practice  it  just  won’t  fly."  We  were  struck  by  the 
extent  of  concern  voiced  by  some  of  the  more  widely  known  and  professional 
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EZ123  Category  checked  (variable  STNRD  1) 
HHIiim  (9)  6.8%  I  I  Coded  from  written  commentary 

](0)  0.0%  (variable  STNRD  2) 

Highly  useful 


Number  of  respondents 


Fig.  20 — Respondents'  opinions  on  standardization 


of  the  respondents  about  standardization  of  many  kinds — of  languages, 
data  formats,  or  documentation.  They  considered  attempts  to  standardize 
premature  and  feared  they  would  stifle  creativity.  Incidentally,  the  few 
respondents  who  favored  standardization  were  mainly  users  with  little 
or  only  recent  experience.  The  relatively  negative  reaction  toward 
creating  regional  centers  was  also  based  on  the  concern  that  such  cen¬ 
ters  would  not  only  add  bureaucratic  problems  but  would  also  separate 
the  work  from  those  who  know,  need,  and  use  it.  Those  who  could  see 
some  merit  in  regional  centers  argued  that  they  would  save  money  on 
personnel  and  computer  resources.  Any  benefit  from  coordinating  data 
processing  and  computer  software  systems  was  mentioned  only  occasionally. 

One  of  the  most  interesting  sets  of  replies  dealt  with  external 
reviewing  procedures  and  boards.  Some  17.4  percent  indicated  that  they 
would  be  highly  harmful;  39.4  percent  suggested  that  they  would  be 
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harmful;  and  only  one  respondent  thought  the  idea  would  be  highly  use¬ 
ful.  The  argument  against  the  review  board  was  that  it  would  be  im¬ 
possible  to  assemble  a  high-quality  panel  to  review  specific  models  in 
sufficient  depth  to  justify  the  time  and  effort.  Many  respondents  felt, 
some  in  no  uncertain  terms,  that  they  were  perfectly  capable  of  provid¬ 
ing  all  the  review  needed.  Based  on  this  survey’s  results  and  informal, 
supplementary  interviews,  we  doubt  that  they  are  right. 

Relationship  to  Other  Descriptors 

We  tabulated  all  opinion  questions  against  the  respondent  s  own 
role,  to  determine  whether  one's  location  in  the  decision  process  had 
any  noticeable  effects,  and  against  the  service  responsible  for  each 
MSG,  to  judge  whether  organizations  had  systematic  preferences. 

While  no  significant  relationships  existed  between  respondent  role 
and  opinions  about  clearinghouses,  external  review,  or  technical  coor- 
dination,  other  matters  were  significant. 

For  example,  the  correlation  of  respondent's  role  with  opinions 
about  standardization  (Table  8)  shows  that  funders /sponsors  most  favor 
the  creation  of  standards,  closely  followed  by  caretakers;  designers 
and  builders  are  the  least  favorable;  and  users  are  somewhere  in  be¬ 
tween.  The  correlation  of  respondent's  role  with  opinions  about  creat¬ 
ing  regional  centers  (Table  9)  reveals  that  funders/sponsors  are  again 
most  in  favor,  although  less  so  than  for  standards.  Again,  designers 
and  builders  are  least  in  favor,  but  they  too  are  less  strongly  com- 
mitted  than  on  the  standardization  issue. 

Tabulations  of  service  opinions  on  the  creation  of  clearinghouses 
are  shown  in  Table  10;  on  standardization,  Table  11;  and  on  technical 
coordination,  Table  12, 

EVALUATION  OF  THE  QUESTIONNAIRE  BY  RESPONDENTS  AND  RESEARCHERS 

We  solicited  respondents'  opinions  about  the  questionnaire,  and 
in  turn  we  rated  the  general  overall  quality  of  each  respondent  s  re¬ 
plies.  Because  the  questionnaire  is  formidable,  we  checked  to  see 
what,  if  any,  systematic  relationships  existed  between  the  time  re¬ 
quired  to  fill  it  out,  the  respondent's  appraisal  of  it,  and  our 


Table  8 


CROSS-TABULATION  OF  RESPONDENT'S  ROLE  AND  OPINIONS  ON  STANDARDIZATION 


Opinion  (Variable  STNRD1) 


Role 

(Variable  RESROL1) 

Highly 

Useful 

Useful 

Same 

Harmful 

Highly 

Harmful 

NR 

Total  (%) 

Funder-Sponsor 

2 

10 

3 

3 

1 

0 

19  (14.4) 

User 

2 

10 

10 

17 

2 

1 

42  (31.8) 

Designer-Builder 

2 

A 

1A 

1A 

5 

2 

41  (31.1) 

Caretaker 

3 

7 

2 

5 

1 

0 

18  (13.6) 

Control 

0 

0 

0 

1 

1 

0 

2  (1.5) 

Other 

0 

1 

1 

1 

A 

3 

10  (7.6) 

Total 

9 

32 

30 

41 

14 

6 

132 

<%) 

(6.8) 

(24.2) 

(22.7) 

(31.1) 

(10.6) 

(4.5) 

(100.0) 

A  AAA  A 

NOTE:  Raw  chi  square  =  56.45,  with  25  deg  freedom.  Significance  -  0.0003 


Table  9 

CROSS-TABULATION  OF  RESPONDENT'S  ROLE  AND  OPINIONS  ON  REGIONAL  CENTERS 


Opinion  (Variable  REGCEN) 


Role 

(Variable  RESR0L1) 

Highly 

Useful 

Useful 

Same 

Harmful 

Highly 

Harmful 

NR 

Total  (%) 

Funder-Sponsor 

0 

5 

1 

8 

A 

1 

19  (14.4) 

User 

1 

A 

5 

21 

5 

6 

42  (31.8) 

Designer-Builder 

0 

6 

3 

18 

9 

5 

41  (31.1) 

Caretaker 

0 

2 

0 

12 

0 

4 

18  (13.6) 

Control 

0 

0 

1 

1 

o  S 

0 

2  (1.5) 

Other 

0 

0 

1 

1 

1 

7 

10  (7.6) 

Total 

1 

17 

ii 

61 

19 

23 

132 

(%) 

(0.8) 

(12.9) 

(8.3) 

(46.2) 

(14.4) 

(17.4) 

(100.0) 

A  Ai  n 

NOTE:  Raw  chi  square  -  42.53,  with  25  deg  freedom.  Significance  -  0.0157 
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Table  10 

CROSS-TABULATION  OF  MILITARY  SERVICE  AND  OPINIONS  ON  CLEARINGHOUSES 


Opinion  (Variable  CLEAR) 


Service 

(Variable 

SERVIC) 

Highly 

Useful 

Useful 

Same 

Harmful 

Highly 
,  Harmful 

NR 

Total  (Z) 

USA 

11 

25 

14 

6 

1 

2 

59  (44.7) 

USAF 

3 

8 

8 

1 

1 

5 

26  (19.7) 

USN 

3 

23 

2 

4 

4 

0 

36  (27.3) 

Other  DOD 

2 

1 

7 

0 

1 

0 

11  (8.3) 

Total 

19 

57 

31 

ii 

7 

7 

132 

(%) 

(14. A) 

(43.2) 

(23.5) 

(8.3) 

(5.3) 

(5.3) 

(100.0) 

NOTE: 

Raw  chi 

square  = 

41.12, 

with  15  deg  freedom. 

Significance  *  0,0003. 


Table  11 

CROSS-TABULATION  OF  MILITARY  SERVICE  AND  OPINIONS  ON  STANDARDIZATION 


Opinion  (Variable  STNRD1) 


Service 

(Variable 

SERVIC) 

Highly 

Useful 

Useful 

Same 

Harmful 

Highly 

Harmful 

NR 

Total  (Z) 

USA 

4 

19 

10 

20 

5 

a 

59  (44.7) 

USAF 

3 

3 

9 

4 

2 

5 

26  (19.7) 

USN 

0 

9 

■a 

13 

7 

0 

36  (27.3) 

Other  DOD 

2 

1 

i 

4 

0 

0 

11  (8.3) 

Total 

9 

32 

30 

41 

14 

6 

132 

(Z) 

(6.8) 

(24.2) 

(22.7) 

(31.1) 

(10.6) 

(4.5) 

(100.0) 

NOTE:  Raw  chi  square  «  35.67,  with  15  deg  freedom. 


Significance  *  0.0020. 


-56- 


Table  12 

CROSS -TABULATION  OF  MILITARY  SERVICE  AND  OPINIONS  ON  TECHNICAL  COORDINATION 


Opinion  (Variable  TECH) 


Service 

(Variable 

SERVIC) 

Highly 

Unde¬ 

sirable 

Unde¬ 

sirable 

Indif¬ 

ferent 

Desirable 

Highly 

Desirable 

NR 

Total  (%) 

USA 

9 

8 

■1 

15 

19 

B 

59  (44.7) 

USAF 

2 

3 

9 

2 

2 

8 

26  (19.7) 

USN 

5 

7 

6 

11 

6 

B 

36  (27.3) 

Other  DOD 

1 

0 

6 

2 

2 

0 

11  (8.3) 

Total 

17 

18 

28 

30 

29 

10 

132 

(%) 

(12.9) 

(13.6) 

(21.2) 

(22.7) 

(22.0) 

(7.6) 

(100.0) 

NOTE:  Raw  chi  square  *  46.85,  with  15  deg  freedom.  Significance  «  0,0001. 


evaluation  of  the  quality  of  the  replies.  We  also  checked  the  two 
evaluation  descriptors  against  the  respondent’s  role  and  the  military 
service  responsible  for  the  MSG. 

We  assessed  the  quality  of  the  responses  before  we  knew  how  the 
respondent  had  evaluated  the  questionnaire.  As  shown  in  Table  13, 
overall  distributions  were  quite  similar.  Though  the  relationship  is 
not  statistically  significant,  users  and  funders /sponsors  were  more 
favorably  disposed  to  the  questionnaire  than  were  others;  designers 
and  builders  were  less  pleased.  We  expected  that  to  some  extent,  be¬ 
cause  the  questionnaire  frankly  favors  issues  of  import  to  the  user 
that  many  builders  apparently  ignore  or  hold  in  less  esteem. 

Our  opinion  of  the  quality  of  respondents’  replies  indicates  that 
users  generally  provided  us  with  better  overall  responses  than  other 
groups,  and  caretakers  provided  the  poorest. 

Service-specific  opinions  were  not  statistically  significant  on 
questions  of  regional  centers,  external  review,  or  quality  of  the  ques¬ 
tionnaire.  Our  own  evaluation  of  responses  by  service  indicates  that 
on  the  average  the  Army  did  better  than  the  others:  15/59  or  24.4  per¬ 
cent  were  good  or  excellent.  Perhaps  that  is  because  of  the  demands 
placed  on  the  Army  segment  of  the  profession  by  the  ad  hoc  Models  Review 
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Table  13 

CROSS -TABULATION  OF  RESEARCHERS’  EVALUATION  OF  RESPONSES 
AND  RESPONDENTS1  EVALUATION  OF  THE  QUESTIONNAIRE 


Respondents’  Evaluation  (Variable  QUEVAL) 


Researchers’ 
Evaluation 
(Variable  IQC) 

Excellent 

Good 

Modal 

I  . . 

Poor 

Bad 

No  Re¬ 
sponse 

Total  (2) 

Excellent 

0 

2 

1 

0 

0 

0 

3  (2.3) 

Good 

2 

7 

12 

3 

1 

1 

26  (19.7) 

Modal 

1 

11 

10 

10 

6 

5 

43  (32.7) 

Poor 

0 

5 

5 

12 

8 

10 

40  (30.4) 

Bad 

0 

2 

6 

7 

2 

3 

20  (15.2) 

Total 

3 

27 

34 

32 

17 

19 

132 

(2) 

(2.3) 

(20.7) 

(25.8) 

(24.3) 

(12.9) 

(14.4) 

(100.0) 

NOTE:  Raw  chi  square  *  53.41,  with  20  deg  freedom.  Significance  *  0.0001. 


Committee.  On  the  other  hand,  other  DOD  and  Air  Force  replies  were 
poor  or  bad,  8/11  or  72.8  percent  and  17/26  or  64,3  percent,  respec¬ 
tively.  Navy  responses  were  predominantly  modal  in  quality,  17/36  or 
47.2  percent. 

Time  taken  to  complete  the  questionnaire  was  significantly  related 
to  both  respondent  and  researcher  evaluations  (see  Table  14) .  Of  the 
30  respondents  who  thought  the  questionnaire  was  excellent  or  good,  26 
or  86.6  percent  required  15  hours  or  less  to  complete  it.  However, 
when  researchers*  evaluation  is  tabulated  against  completion  time 
(Table  15),  the  excellent  and  good  responses,  while  concentrated  under 
15  hours,  are  spread  somewhat  more  along  the  time  dimension.  We  con¬ 
jecture  that  if  a  respondent  intimately  knew  his  MSG,  he  could  respond 
thoroughly  and  competently  in  under  15  hours;  if  not  so  knowledgeable 
but  sympathetic  to  the  purposes  of  the  investigation,  he  could  produce 
high-quality  responses  at  some  additional  cost  in  time.  It  is  also  in¬ 
teresting  to  see  that  as  regards  the  three  respondents  who  took  over 
50  hours  to  complete  the  questionnaire,  mutual  enmity  prevails:  they 
rated  the  questionnaire  low,  and,  despite  their  protracted  completion 
time,  we  found  their  answers  poor. 


-58- 


Table  14 

CROSS-TABULATION  OF  RESPONDENTS’  COMPLETION  TIME 
AND  EVALUATION  OF  THE  QUESTIONNAIRE 


Evaluation  (Variable  QUEVAL) 


Time 

(Variable  QT) 
(hr) 

Excellent 

Good 

Modal 

Poor 

Bad 

No  Re¬ 
sponse 

Total  (%) 

0-5 

2 

ii 

9 

21 

8 

10 

61  (46.1) 

6-10 

0 

6 

10 

8 

4 

3 

31  (23.5) 

11-15 

0 

7 

6 

0 

0 

1 

14  (10.6) 

16-20 

1 

2 

3 

0 

1 

0 

7  (5.3) 

21-25 

0 

0 

3 

0 

0 

1 

4  (3.0) 

26-30 

0 

0 

1 

1 

0 

1 

3  (2.3) 

31-35 

0 

0 

0 

1 

2 

0 

3  (2.3) 

36-40 

0 

0 

1 

1 

0 

0 

2  (1.6) 

41-45 

0 

0 

0 

0 

0 

0 

0  (0.0) 

46-50 

0 

1 

0 

0 

0 

0 

1  (0.8) 

2:  51 

0 

0 

0 

0 

2 

1 

3  (2.3) 

NR 

0 

0 

1 

0 

0 

2 

3  (2.3) 

Total 

(%) 

3 

(2.3) 

27 

(20.7) 

34 

(25.8) 

32 

(24.3) 

17 

(12.9) 

19 

i  (14.4) 

132 

(100.0) 

NOTE :  Raw 

chi  square 

=  82.64,  with  55  deg  freedom.  Significance  -  0.0125 
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Table  15 

CROSS-TABULATION  OF  QUESTIONNAIRE  COMPLETION  TIME 
AND  RESEARCHERS'  EVALUATION  OF  RESPONSES 


Evaluation  (Variable  IQC) 


Time 

(Variable  QT) 
(hr) 

Excellent 

Good 

Modal 

Poor 

Bad 

Total  (%) 

0-5 

0 

7 

20 

22 

12 

61  (46.1) 

6-10 

2 

5 

15 

6 

3 

31  (23.5) 

11-15 

1 

1 

3 

2 

1 

14  (10.6) 

16-20 

0 

0 

4 

1 

2 

7  (5.3) 

21-25 

0 

3 

0 

1 

0 

4  (3.0) 

26-30 

0 

1 

0 

2 

0 

3  (2.3) 

31-35 

0 

1 

0 

2 

0 

3  (2.3) 

36-40 

0 

0 

1 

1 

0 

2  (1.6) 

41-45 

0 

0 

0 

0 

0 

0  (0.0) 

46-50 

0 

1 

0 

0 

0 

1  (0.8) 

£  51 

0 

0 

0 

2 

1 

3  (2.3) 

NR 

0 

1 

0 

1 

1 

3  (2.3) 

Total 

3 

26 

43 

40 

20 

132 

(%) 

(2.3) 

(19.7) 

(32.7) 

(30.4) 

(15.2) 

(100.0) 

NOTE:  Raw  chi  square  =  61.03,  with  44  deg  freedom. 
Significance  =  0.0512. 
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III.  CONCLUSIONS 


The  general  discipline  known  as  gaming/simulation  is  highly  diverse. 
It  encompasses  at  least  four  different  subjects  that  have  different 
criteria  for  validation  and  different  measures  for  cost  and  effective¬ 
ness.  That  makes  it  a  difficult  entity  to  grasp  analytically.  Never¬ 
theless,  in  the  following  pages,  we  venture  our  opinions  on  the  sig¬ 
nificance  of  the  results  of  our  survey.  They  are  summarized  under  the 
same  substantive  headings  as  were  the  results  analyzed  in  Sec.  II. 

MSG  PURPOSES 

Weapon  system  evaluations,  mostly  built  by  the  individual  military 
services  for  their  own  exclusive  purposes  and  use,  predominate,  both 
in  absolute  level  of  activity  and  in  total  expenditures.  Ironically, 
expenditures  and  public  knowledge  about  gaming  and  simulation  are  in¬ 
versely  related.  Free-fonn  gaming  has  received  great  and  continuing 
notoriety,  but  its  expenditures  are  trifling  compared  with  expenditures 
on  all-machine,  technical  evaluations.  Many  models,  simulations,  and 
games  are  literally  unknown  outside  a  small  user-producer  coterie. 

The  degree  of  knowledge  that  anyone  of  even  this  in-group  may  have 
about  a  particular  MSG  is  evidently  limited.  Many  poor,  incomplete, 
and  low-confidence  replies  were  obtained  from  responsible  and  knowledge¬ 
able  professionals.  That  several  respondents  took  upwards  of  70  hours 
to  complete  the  questionnaire  reinforces  this  view. 

Also  worrisome  is  the  amount  of  what  is  essentially  research  money 
that  is  being  spent  in  the  absence  of  rigorous  and  accepted  research 
standards.  As  a  result,  basic  knowledge  about  both  substantive  and  pro¬ 
cedural  matters  is  neglected.  Very  little  is  being  spent  on  validation 
efforts  and  on  basic  research  about  MSG  methods,  data,  and  uses,  with¬ 
out  which  the  credence  of  gaming  and  simulation  suffers. 

In  our  opinion,  the  copious  in— house  production  of  technical— eval¬ 
uation  models  is  being  pursued  without  sufficient  attention  to  quality 
and  scientific  rigor: 
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o  In  many  cases,  in-house  work  has  ignored  scientific  standards 
of  data  collection,  management,  and  validation, 
o  Documentation  of  in-house  work  is  often  very  poor;  this  fail¬ 
ing  is  frequently  rationalized  on  the  grounds  that  the  work  is 
not  meant  to  get  outside  the  builder’s  shop, 
o  About  half  the  sampled  MSGs  were  not  externally  reviewed;  most 
respondents  reject  the  need  for  such  reviews  on  bureaucratic 
rather  than  scientific  grounds. 

PRODUCTION 

The  evident  preference  for  large,  all-machine  models  and  simula¬ 
tions  is  questionable  on  several  grounds.  Large-scale,  finely  detailed 

MSGs  that  try  to  deal  with  problems  having  significant  uncertainties 

* 

may  only  serve  to  generate  errors,  not  clarifying  anything.  Given 
what  appear  to  be  weak-to-poor  data,  extremely  fine  temporal  and  spatial 
levels  of  model  resolution,  and  low  levels  of  demonstrated  concern  for 
supporting  research,  the  MSGs  produced  may  have  doubtful  reliability. 

Large  models  are  usually  complicated,  expensive  to  build  and  use, 
take  extended  periods  to  operate  and  interpret,  and  are  the  least  sci¬ 
entifically  defensible.  They  quickly  begin  to  suffer  from  the  disor¬ 
ganization  created  by  changes  in  purpose  and  personnel,  bad  documenta¬ 
tion,  gaps  in  logic,  and  problems  of  data-base  preparation,  maintenance, 
and  validation. 

If  large  models  must  be  produced,  the  key  to  control  seems  to  be 
in  continuity  of  personnel.  Changes  of  personnel  have  significant  ef¬ 
fects.  Usage  decreases  because  no  one  knows  what  a  model  is  supposed 
to  do,  how  it  does  it,  or  why.  Where  sunk  costs  are  great,  there  is  a 
tendency  to  use  a  large  and  expensive  model  anyway,  even  though  none 
of  its  caretakers  can  determine  its  validity  for  new  applications.  Doc¬ 
umentation  should  ameliorate  this  problem,  but  it  seldom  does. 

*For  a  technical  discussion  see  John  W.  Tukey,  "The  Propagation  of 
Errors,  Fluctuations  and  Tolerances:  Basic  Generalized  Formulas,1’  Re¬ 
port  No.  10,  Department  of  Mathematics,  Princeton  University;  for  a 
more  practical  view,  see  William  Alonso,  "The  Quality  of  Data  and  the 
Choice  and  Design  of  Predictive  Models,"  in  G.  C.  Hemmens,  ed.,  Urban 
Development  Models ,  Special  Report  No.  197,  Highway  Research  Board, 
Washington,  D.C.,  1968,  pp.  178-192. 
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OPERATIONS 

We  believe  that  documentation  is  considerably  worse  than  the  ques¬ 
tionnaire  replies  indicate.  Documentation  standards,  requiring  that 
documentation  .meet  certain  specific  criteria,  cost  little  extra  for  their 
large  contribution  to  better  management  control.  The  need  is  particu¬ 
larly  great  in  the  case  of  very  large  simulations.  When  generations  of 
programmers  perpetuate  inconsistencies  and  errors  in  sloppily  updated 
documentation,  all  can  be  lost. 

Many  capabilities  built  into  these  devices  have  not  been  subjected 
to  validation.  Not  only  is  their  empirical  basis  dubious  or  admitted 
to  be  lacking,  but  few  efforts  are  being  made  to  collect  missing  or 
questionable  input  data  or  to  execute  sensitivity  analyses  according  to 
an  appropriate  experimental  design.  The  lack  of  sensitivity  analysis 
is  related  to  deficiencies  in  estimating  the  validity  of  input  param¬ 
eters.  Neither  of  these  matters  seems  to  be  taken  seriously.  There 
is  a  less  than  50/50  chance  that  a  sensitivity  test  will  be  done,  and 
when  it  is  done,  there  is  frequently  no  record  of  the  outcome. 

Most  of  the  surveyed  MSGs  have  not  been  subjected  to  any  external 
review,  with  the  result  that  many  contain  implicit  and  intangible  in¬ 
puts  whose  existence  and  rationale  have  not  been  documented  in  any  way. 
The  so-called  "ins titutional  memory"  in  the  general  system  is  not  very 
well  developed.  Even  about  MSGs  of  fairly  recent  vintage,  respondents 
seem  unable  to  answer  technical  and  cost  questions  very  well  or  with 
much  confidence. 

One  colleague  has  suggested  only  half  facetiously  that  validation 
is  a  happy  customer.  If  so,  is  ultimate  validation  a  follow-on  con¬ 
tract?  A  comprehensive  review  of  data-validity  problems  is  needed,  and 
some  strong  corrective  measures  should  be  instituted.  Advocacy  rather 
than  scientific  preferences  seems  to  predominate. 

USE 

In  our  opinion,  the  level  of  professional  communication  is  danger¬ 
ously  low.  There  is  a  great  need  for  better  coordination,  documentation, 

*J.  P.  Mayberry,  "Principles  for  Assessment  of  Simulation  Model  Va¬ 
lidity,"  in  A.  I.  Siegel,  ed.,  Proceedings  of  the  Symposium  on  Computer 
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and  studies  of  use  at  the  operational,  experimental,  and  administrative 
interfaces.  It  is  not  enough  that  a  study  be  finished  according  to  for¬ 
mal  contract  specifications;  what  becomes  of  the  study  and  how  it  is 
used  are  far  more  important  considerations.  Learning,  both  kinds  and 
amounts,  must  be  evaluated  so  that  resources  can  be  expended  more  ra¬ 
tionally.  Perhaps  it  would  be  worthwhile  to  declare  a  moratorium  on 
expenditures  for  new  studies  until  existing  ones  have  been  properly 
evaluated.  Technical  weapon  evaluations  that  are  either  unused,  little 
used,  or  misused,  for  example,  may  be  worse  than  no  studies  at  all.  Im¬ 
prudent  management  policies  (for  instance,  the  high  regular  turnover  of 
key  personnel  in  some  military  activities) ,  coupled  with  uneven  documen¬ 
tation  standards  and  procedures,  may  account  for  the  ineffective  use 
of  models.  If  no  one  remembers  why  a  particular  model  was  built,  for 
whom,  and  how  it  operates,  it  may  be  used  incorrectly,  or,  worse,  peo¬ 
ple  may  start  from  scratch  to  build  a  model  that  will  do  the  job  it 
was  in  fact  designed  to  do.  If  there  is  little  or  no  documentation, 
the  potential  waste  becomes  a  practical  certainty. 

It  is  difficult  to  determine  what  influence  these  MSGs  have  had. 

As  noted,  half  of  them  did  not  even  result  in  a  direct  briefing.  Writ¬ 
ten  comments  roughly  suggest  that  they  "did  the  job";  however,  little 
information  was  given  on  the  importance  of  the  "job"  and  the  policy 
decisions  that  depended  upon  it.  Documentation  accompanying  any  MSG 
should  indicate  what  it  has  been  used  for,  who  has  used  it,  when  it  was 
used,  and  so  forth. 

The  dearth  of  written  scenarios  and  explicit  consideration  of  in¬ 
tangible  and  uncertain  elements  is  of  concern.  It  is  all  too  easy  to 
misuse  well-specified  numerical  models  by  changing  the  problem  context 
or  interpretation.  A  model  that  is  impressively  "good  in  one  context 
may  be  inappropriate  in  another  context,  even  if  specific  data  for  the 
hard  numbers  are  adequate.  That  depends  upon  the  skills  and  inter¬ 
face  of  those  who  know  the  original  uses  of  the  model,  those  who  de¬ 
termine  the  new  purposes  and  applications,  those  who  set  the  soft  num¬ 
bers,  and  those  who  provide  the  background  interpretations.  There 

Simulation  as  Related  to  Manpower  and  Personnel  Planning ,  Naval  Per¬ 
sonnel  R&D  Laboratory,  Washington,  D.C.,  July  1971,  pp.  157-165. 
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is  little  evidence  on  how  those  tasks  are  being  done,  if  at  all.  It  ap 
pears  that  the  engineering,  applied-science  bias  of  most  of  the  builders 
and  caretakers  we  surveyed  would  make  replication  of  their  MSGs  risky. 

What  all  of  this  means  can  be  stated  briefly:  there  is  no  substi¬ 
tute  for  people  who  know  their  business.  One  of  the  real  dangers  that 
we  perceive  is  the  poverty  of  the  scientific  interface  between  those 
who  know  their  business  and  those  who  are  trying  to  generate  business. 

The  Department  of  Defense  has  nurtured  a  group  of  specialists  who, 
having  made  their  share  of  mistakes,  are  doing  some  competent  and  use¬ 
ful  work.  But  there  are  few  communication  means  by  which  their  knowl¬ 
edge  can  be  shared  more  broadly  with  those  in  the  civil  sector. 

Notwithstanding  this  emergent  professionalism,  the  tenuousness  of 
much  of  the  data  being  used,  the  immature  extent  and  level  of  valida¬ 
tion,  and  the  relative  neglect  of  such  important  scientific  and  opera¬ 
tional  procedures  as  sensitivity  analysis  and  scrutiny  of  the  appropri¬ 
ateness  of  work  for  specific  operational  environments  and  scenarios, 
make  it  easy  to  infer  that  advocacy  rather  than  scientific  preferences 
prevails.  Can  scientific  content  be  improved  and  unfounded  advocacy  be 
reduced?  Generally,  the  answer  is  yes,  although  the  problem  is  diffi¬ 
cult. 

One  desirable  innovation  would  be  to  make  the  advocacy  process  two- 
sided  rather  than  one-sided  as  it  is  now.  For  example,  the  quality 
of  discourse  on  weapons  procurement  would  be  improved  if  congressional 
participants  as  well  as  DOD  proponents  had  consultant  advice  and  pro¬ 
fessional  assistance.  In  other  words,  improve  existing  methods  and 
procedures  for  challenging  the  assumptions  and  quality  of  work  done  in 
support  of  any  one  position,  system,  or  decision.  Lawyers  and  accoun 
tants  may  be  able  to  win  debating  points  from  the  engineers,  generals, 
and  mathematicians,  but  that  suggests  a  far  from  rational  allocation 
process,  much  less  an  optimal  one. 

*The  recent  discussions  between  George  Rathjens  and  Albert 
Wohlstetter  on  the  proposed  Safeguard  ABM  system  before  the  U.S.  Senate 
illustrate  the  benefits  of  having  both  sides  on  a  question  technically 
competent  and  informed  with  comparable  data.  The  entire  issue  on  Op¬ 
erations  Research ,  Vol.  19,  September  1971,  is  devoted  to  their  intelli¬ 
gent  debate. 
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COSTS 

Costing  is  deficient.  Granted,  cost  accounting  is  difficult  to 
do  in  a  process  as  complex  as  the  initiation,  production,  operation, 
and  evaluation  of  MSGs;  nevertheless,  for  better  management  control, 
some  record  of  cost  must  be  kept  on  each  MSG.  It  need  not  be  compli¬ 
cated;  even  crude  figures  within  ±100  percent  of  the  actual  would 
be  an  improvement  over  current  information. 

More  than  one-third  of  the  respondents,  over  a  range  of  diverse 
questions,  did  not  or  could  not  consistently  reply  to  the  most  elemen¬ 
tary  cost  questions.  Those  that  did  respond,  moreover,  had  low  con¬ 
fidence  in  their  answers. 

Whose  responsibility  and  in  whose  interest  is  cost  control?  De¬ 
velopers  have  little  reason  to  be  interested  in  costs  except  as  they 
contribute  to  their  own  revenues.  Questions  about  alternative  methods 
to  modeling  and  simulation  are  seldom  explored  in  the  operational  set¬ 
ting,  Current  procedures  do  not  seem  to  include  formal  consideration 
of  whether  there  is  a  cheaper,  easier  way  to  proceed  or  whether  model¬ 
building  is  really  the  most  appropriate  technique.  The  first  should  be 
a  managerial  concern;  the  second  should  be  dealt  with  at  the  technical 
level. 

That  responsible  persons  were  unable  to  supply  much  rudimentary 
cost  data  suggests  that  cost  accounting  has  been  neglected.  It  is 
evident  that  cost  data,  perhaps  conceived  of  primarily  as  relating  to 
investment,  quickly  become  separated  from  the  work  itself.  What  re¬ 
sults  is  that  users,  caretakers,  and  even  builders  have  but  vague  no¬ 
tions  of  cost  a  year  or  so  after  the  work  has  been  undertaken. 

Curiously,  there  appears  to  be  a  magic  (i.e.,  relatively  easily 
fundable)  budget  amount  between  $200,000  and  $300,000.  Judging  from 
our  survey  data,  it  would  be  better  to  apply  for  two  separate  budgets 
for  two  different  models  at  $200, 000-$300, 000  each  than  to  apply  for  a 
budget  for  one  model  at  $400,000-$600,000.  Families  of  MSGs  have  been 
maintained  for  years  with  separately  labeled  components  funded  sepa¬ 
rately  and  used  and  evaluated  independently.  Actually,  many  are  merely 
increments  of  the  same  basic  work  and  should  be  considered  as  such. 


-66- 


It  is  interesting  that  most  money  is  spent  where  professional 
visibility  and  active  participation  by  high— echelon  personnel  are  mini¬ 
mal.  That  raises  an  important  question  about  the  effectiveness  of  ex¬ 
penditures.  Are  we  trying  to  compare  incommensurables? 

Machine  models  and  simulations  appear  to  favor  so-called  value- 
free  engineering  work  and  are  produced  by  bright  lieutenants,  captains, 
majors,  or  possibly  colonels  in  cooperation  with  civilian  contractors. 
Such  work  can  easily  lead  to  larger  studies  that  become  briefings  for 
colonels  and  generals  and  their  civilian  counterparts.  Information 
about  the  context,  such  as  verbal  descriptions  and  scenarios,  and  de¬ 
tails  about  purposes  and  limitations  of  a  model  are  seldom  spelled  out. 
Man-machine  exercises,  in  contrast,  are  frequently  used  for  teaching  or 
training  in  the  staff  colleges.  As  a  result,  there  is  some  chance  that 
a  two-  or  three-star  general  or  admiral  might  remember  or  have  learned 
something  as  a  result  of  a  man-machine  exercise  he  participated  in. 

Such  activities  may  also  be  used  in  an  experimental  laboratory  where 
the  personnel  employed  are  not  necessarily  military  and  where  the  pur¬ 
pose  is  altogether  different.  As  noted,  the  least  expensive  activities 
are  the  political-military  exercises  that  at  one  time  or  another  may 
have  commanded  the  attention  of  the  highest-level  personnel.  Whether 
free-form  military  games  accomplish  anything  besides  entertainment  is 
an  open  question  that  deserves  further  study. 

Little  attention  has  been  paid  to  what  is  meant  by  claims  that  a 
certain  study  has  influenced  policy.  An  imaginative  briefing  by  the 
likes  of  Herman  Kahn  or  a  political-military  exercise  run  by  an  Albert 
Wohlstetter  with  high-level  participants  may  have  had  more  influence 
on  policy  than  most  multi— million— dollar  models  and  simulations.  Then 
again,  neither  may  make  any  difference.  Much  depends  on  the  timeliness 
of  the  exercise  and  its  relevance  to  current  problems. 

Questions  like  the  foregoing  must  be  made  explicit.  Then  it  may 
be  possible  to  take  a  more  objective  look  at  the  routine  expenditure 
of  millions  of  dollars  for  middle-level,  engineering- type  MSGs. 
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IV.  RECOMMENDATIONS 


From  our  analysis  of  the  survey  results,  we  have  identified  the 
following  issues  as  being  critical  to  resolve  if  the  standards  of 
gaming  and  simulation  are  to  be  raised. 

ADVOCACY  VERSUS  SCIENTIFIC  VALIDATION 

The  process  of  building  and  using  MSGs  provides  an  important  illus¬ 
tration  of  model-builders'  neglect  of  science  in  the  interest  of  advocacy. 
As  long  as  model-builders  do  not  question  the  environment  set  for  them 
by  those  soliciting  the  work,'  practically  any  point  of  view  can  be  sup¬ 
ported  by  selecting  appropriate  "guesstimates"  about  the  environment. 
Accordingly,  there  is  a  need  for  open,  regular,  and  more  rigorous  review 
of  the  models,  games,  and  simulations  that  are  being  built  and  used. 
Procedures  should  challenge  the  validity  of  data  inputs  that  are  now  so 
routinely  and  unquestioningly  used. 

Communication  between  civilian  and  military  segments  of  the  pro¬ 
fession  must  be  improved.  A  professional  advisory  group  at  the  level 
of  Congress  or  the  GAO  appears  to  be  desirable  to  enhance  the  dialogue 
between  the  Congress  and  DOD. 

COST  ACCOUNTING 

Given  the  little  or  no  institutional  memory  about  the  cost  of  build¬ 
ing,  running,  updating,  and  evaluating  MSGs,  any  effort  would  probably 
be  an  improvement  over  the  present  poor-to-nonexistent  procedures  of 
cost  accounting.  The  accounting  definitions  and  categories  we  were 
forced  to  create  for  this  survey  may  be  useful  as  a  point  of  departure. 

At  the  least,  a  brief  cost  dossier  should  be  kept  with  an  MSG  throughout 
its  existence. 

EXTERNAL  REVIEW  AND  PROFESSIONAL  STANDARDS 

Professional  reviewing  is  a  critical  and  much-neglected  means  of 
quality  control.  To  those  who  claim  that  it  merely  adds  an  extraneous 
bureaucratic  impediment  to  "getting  the  job  done,  we  cite  the  efficient 


and  competent  efforts  of  the  Army  Models  Review  Committee  as  evidence 
£ 

to  the  contrary.  Regrettably,  that  committee’s  work  was  ad  hoc  and 
temporary,  when  persistence  is  called  for. 

The  Joint-  Chiefs  of  Staff  might  well  be  able  to  assume  an  MSG  re¬ 
view  function  for  all  the  services.  What  is  required  are  qualified  pro¬ 
fessionals  to  serve  on  a  long-term,  continuing  basis  and  strong  incen¬ 
tives  to  insure  adherence  to  scientific  standards  of  evaluation.  Such 
a  group  should  also  deal  with  questions  of  standards,  such  as.  When  is 
sensitivity  analysis  needed?  and  What  constitutes  data  validation? 

DOCUMENTATION 

Documentation  is  largely  of  uneven  quality,  not  available,  or  non¬ 
existent.  At  a  minimum,  an  MSG’s  documentation  should  include  the 
following:  the  program  listing;  flow  charts;  variable  listings,  defi¬ 

nitions,  and  sources;  the  program  deck  with  comments  and  caveats  about 
operating  quirks  and  special  library  or  input/output  routines;  the  op¬ 
erator’s  manual;  the  programmer’s  manual;  the  player’s  manual  if  the 
MSG  is  man— machine  or  free— form;  the  pertinent  analysis  routines  used 
to  reduce  data  generated  by  the  MSG  and  to  estimate  input  parameter 
values  for  the  MSG;  appropriate  data  reflecting  what  the  MSG  cost  to 
construct,  update,  and  run;  and  the  register  of  critical  personnel  in¬ 
volved  in  MSG  initiation — who  wanted  it  built,  for  what  reasons;  pro¬ 
duction — the  identities  of  the  master  modeler  and  the  model  team  and 
what  validation  procedures  they  used;  operations — the  history  of  pro¬ 
fessional  review  by  persons  external  to  the  builder— user ;  and  use  who 
used  it,  when,  and  with  what  purpose  and  outcome. 

Many  will  complain  that  this  list  is  too  long,  is  burdensome,  and 
would  impede  the  work  at  hand.  Nevertheless,  conscientious  and  con¬ 
sistent  attempts  to  complete  it  will  improve  existing  management  prac¬ 
tices  and  may  in  time  open  the  way  to  improvements  in  the  state  of  the 
art  and  the  state  of  knowledge. 

*Army  Models  Review  Committee,  Review  of  Selected.  Army  Models ,  De- 
partment  of  the  Army,  Washington,  D.C.,  May  1971,  Chapter  8. 
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REDUNDANCY  AND  STANDARDIZATION 

We  find  no  evidence  that  redundancy  is  excessive.  Communication 
amoung  MSG  builders  is  poor  and  needs  improvement 5  however,  even  if  it 
were  good,  a  certain  amount  of  redundancy  would  be  desirable,  and  that 
amount  does  not  appear  to  have  been  consistently  exceeded. 

Standardization  should  not  be  confused  with  adherence  to  profes¬ 
sional  standards.  We  recommend  strongly  against  the  first,  as  it  is 
premature.  We  urge,  however,  that  efforts  be  made  to  promote  the  second* 

A  CENTRAL  CLEARINGHOUSE? 

We  recommend  against  the  creation  of  an  additional  bureaucratic 
operation  for  the  clearing  of  work  on  MSGs.  However,  there  is  a  le¬ 
gitimate  need  for  a  professional  focal  point,  which  could  be  provided 
by  the  type  of  JCS  professional  review  board  recommended  above. 

Work  in  gaming  and  simulation  is  fragmented  to  an  unnecessary  and 
unhealthy  extent.  We  urge  the  joint  usage  of  MSGs.  By  that  we  mean 
the  sharing  of  MSG  construction  and  use  among  the  military  services  and 
government  agencies;  the  scientific  replication  of  MSGs;  collective  at¬ 
tempts  to  account  for  and  reduce  costs;  and  the  sharing  of  special  over¬ 
head  charges  for  large-scale,  general-purpose  computational  systems, 
even  more  than  is  now  done  via  the  NMSCCS  device.  Joint  usage  promises 
the  more  efficient  use  of  scarce  professional  talent  and  the  diffusion 
of  professional  standards.  The  creation  of  a  JCS  Models  Review  Commit¬ 
tee  based  on  the  Army  prototype  would  be  a  step  in  the  right  direction. 


RESEARCH 

Basic  research  and  knowledge  is  lacking.  The  majority  of  the  MSGs 
sampled  are  living  off  a  very  slender  intellectual  investment  in  funda¬ 
mental  knowledge.  While  this  is  probably  not  the  time  to  expect  much 
funding  for  basic  research,  the  need  is  great  for  work  on  topics  such 
as  simulation  methods,  data  validation,  sensitivity  analysis,  and  sta¬ 
tistical  tests  for  simulation  outputs.  As  regards  analysis,  study  is 
needed  on  formal  models  of  combat,  such  as  allocation  and  search  game 
theory.  In  the  "softer”  subjects  that  bear  directly  on  applied  MSGs, 
there  is  a  need  for  studies  of  panic  behavior  (the  breaking  point 


-70- 


hypothesis,  for  example) ,  threat  and  confrontation,  and  especially  hu¬ 
man  factors  and  motivation.  At  another  level,  work  is  required  on 
basic  questions  of  use,  both  for  particular  MSGs  and  for  whole  families 
and  classes  of  MSGs. 

The  "image"  of  research  needs  refurbishment  among  funders  and 
builders  in  the  professional  community.  Research  appears  to  be  so  stig¬ 
matized  that  one  can  scarcely  acknowledge  sponsorship  of  a  pure  research 
project  without  bracing  for  criticism. 

The  need  for  basic  research  is  so  critical  that  if  no  other  fund¬ 
ing  were  available  we  would  favor  a  plan  to  reduce  by  a  significant 
proportion  all  current  expenditures  for  MSGs  and  to  use  the  saving  for 
basic  research. 

MSG  SIZE 

There  is  every  indication  that  the  larger  MSGs  have  been  of  little 
utility.  The  size,  length  of  time  under  development,  and  generality  of 
an  MSG  all  appear  to  be  directly  related  to  the  difficulty  of  control¬ 
ling,  validating,  and  using  it.  Undesirable  outcomes  resulting  from 
changes  in  personnel,  bad  documentation,  poor  conceptualization,  and 
poor  professional  communication  and  review  are  only  exaggerated  with 
large  MSGs.  We  recommend  that  standards  for  approving  the  construction 
of  large-scale  MSGs  be  much  more  stringent  than  for  smaller  projects. 

We  believe  that  large-scale  MSGs  tend  to  lack  the  capability  of 
handling  scenarios  and  other  hard-to-quantify  elements.  Funds  would  be 
better  spent  on  the  basic  research  to  acquire  that  capability  than  on 
the  premature  constuction  of  large  programs. 

FREE-FQRM  AM)  MAN-MACHINE  GAMING 

We  believe  that  it  is  time  for  an  assessment  (preferably  by  mili¬ 
tary  and  nonmilitary  personnel,  "hard"  and  "soft"  scientists)  of  the 
role  of  free-form  and  man-machine  gaming  in  DOD  work  on  MSGs,  relative 
to  that  of  all-machine  simulation.  We  suspect  that  free-form  gaming 
deserves  more  prominence. 

Man-machine  gaming  for  operational  purposes  appears  to  be  rela¬ 
tively  expensive  and  not  heavily  used.  For  training  and  teaching, 
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both  man-machine  and  free-form  gaming  appear  to  have  been  relatively 
undersupported. 

We  recommend  that  an  investigation  be  sponsored  to  explore  the  re¬ 
lationship  among  multiple  uses  of  MSGs.  Specifically,  especially  given 
the  projected  size  of  the  new  Naval  War  College  gaming  facility,  the 
relationship  among  training,  evaluation,  operational,  and  research  uses 
must  be  better  understood. 


GAMING  IN  THE  CIVILIAN  SECTOR 

In  spite  of  some  negative  findings,  we  see  distinct  growth  in  the 
state  of  the  art  and  the  emergence  of  professional  standards.  We  are 
troubled,  however,  by  the  lack  of  professional  communication  between 
and  outside  the  specialized,  in-house  model-building  shops.  It  seems 
likely  that  many  of  the  mistakes  committed  in  the  application  of  sim¬ 
ulation  and  gaming  to  DOD  problems  are  about  to  be  re-committed  in 
civil-sector  applications.  Given  the  lack  of  communication  that  sug¬ 
gests,  it  may  be  fortunate  that  funding  for  MSGs  from  agencies  such  as 
HEW  and  HUD  is  small  relative  to  that  from  DOD. 

””  £ 

Indeed  two  pioneering  attempts  by  HUD  have  been  appraised  and 
found  wanting  in  many  of  the  same  ways  as  the  MSGs  we  report  on  here. 
The  problems  are  apparently  common.  See  Garry  D.  Brewer,  The  Politi¬ 
cian  ^  the  Bureaucrat y  and  the  Consultant:  A  Critique  of  Urban  Problem- 
Solving,  New  York:  Basic  Books,  forthcoming  in  1973. 


I 
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Appendix  A 

QUESTIONNAIRE:  MODELS,  COMPUTER  MACHINE  SIMULATIONS, 
GAMES,  AND  STUDIES* 


PREFACE 


The  purpose  of  this  questionnaire  is  to  aid  the  interested  profes¬ 
sional  in  describing,  characterizing,  and  analyzing  his  game,  model,  or 
simulation.  It  is  a  first  and,  hopefully,  useful  step  in  the  clarifica¬ 
tion  of  professional  standards  in  the  work  on  gaming  and  simulation. 
Furthermore,  the  questionnaire  is  designed  so  that  it  might  also  serve 
as  a  device  for  communicating  and  cataloging  different  games  and  simula¬ 
tions  in  a  format  that  encourages  easy  interchange  of  information. 

The  questionnaire  format  has  been  adopted  for  three  purposes. 

(1)  In  the  course  of  our  ongoing  investigations  we  expect  to  use  this 
document  as  a  questionnaire.  (2)  Stress  in  design  has  been  to  produce 
a  categorization  scheme  for  the  description  and  classification  of  games 
and  simulation  in  general.  The  goals  are  to  help  establish  professional 
standards  and  to  explore  the  possibilities  of  developing  a  reasonably 
good  classification  and  consistent  description  that  covers  many  games  of 
different  varieties.  (3)  It  is  our  belief  that  the  compiling  of  a  large 
consistent  sample  of  many  games  and  simulations  for  the  purposes  of  anal 
ysis ,  evaluation,  information  interchange,  and  the  construction  of  pro¬ 
fessional  standards  is  overdue.  The  work  involved  in  doing  so  is  both 
large  and  onerous.  The  handling  of  large  quantities  of  data  calls  for 
at  least  partially  computerized  procedures.  The  format  we  are  pre¬ 
senting  here  was  designed  with  this  type  of  data  processing  in  mind. 

*This  is  the  survey  instrument,  reproduced  in  its  entirety.  (It 
was  previously  published,  by  the  same  authors  under  the  same  title ,  as 
P-4672,  The  Band  Corporation,  July  1971.)  Page  numbers  have  been 
changed  to  be  in  sequence  with  this  report .  Responses  to  many  of  the 
questions  in  Parts  I  and  II  are  indicated  in  italics. 
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This  questionnaire  had  as  an  original  purpose  the  description  and 
classification  of  games  in  general.  Because  the  preponderance  of  all 
available  resources  for  models,  simulations,  and  games  is  spent  by  the 
Defense  Department  and  its  various  derivates,  we  have  redesigned  and 
reorganized  the  instrument  to  reflect  these  activities  better.  Some 
questions  were  reworded,  some  were  added,  others  were  discarded.  At 
this  point  in  the  process,  separate  questionnaires  for  computer,  all¬ 
machine  models,  and  for  man— machine  and  manual  games  were  constructed* 

These  modified  questionnaires  were  applied  to  ten  representative 
DOD  models,  simulations,  and  games.  Several  Rand  Corporation  games 
were  also  sampled.  This  "shakedown"  prompted  another  round  of  evalu¬ 
ation  and  redesign.  The  present  version  of  a  single  questionnaire  is 
the  result.  Although  the  emphasis  on  DOD  is  evident,  we  believe  that 
a  hard  core  of  generally  applicable  information  remains. 

Questionnaire  conceptualization  and  design  is  a  potentially  diffi¬ 
cult  undertaking.  This  particular  questionnaire  covers  a  complex,  diverse, 
changing,  and  specialized  body  of  knowledge  and  practice;  consequently, 
the  difficulties  found  in  its  construction  have  been  formidable.  For 
instance,  even  at  the  most  primitive  definitional  level  there  is  little 
but  cloying  disagreement.  What  is  a  model?  What  difference  is  there 
between  a  model  and  a  simulation?  What  is  a  game  and  when  is  it  not  a 
model,  and  vice  versa?  Indeed  the  semantic  game  presently  appears  to 
take  precedence  and  to  substitute  for  the  real  game  all  too  frequently. 
Other  indicators  of  an  unsettled  but  emerging  professionalism  abound. 

What  this  means  is  that  construction  of  a  questionnaire  such  as  this  is 
hard  work;  and  no  matter  what  results,  it  will  have  shortcomings,  prob¬ 
lems,  and  probably  more  than  a  few  errors.  We  acknowledge  the  weaknesses 
and  welcome  comments,  suggestions,  and  corrections  from  others  concerned 
with  gaming  and  simulation. 

Parts  I,  II,  and  IV  are  designed  for  all  types  of  models,  games,  and 
simulations.  Manual  and  man-machine  activities  necessitated  the  addi¬ 
tional  questions  in  Part  III  as  well.  Further  work  is  in  progress  in  the 
categorization  of  questions  more  specifically  aimed  at  the  uses  of  gaming 
and  simulation  for  teaching  and  for  experimental  purposes.  We  recognize 
that  this  questionnaire  cannot  be  regarded  as  adequate  in  providing  a 
means  of  analysis  if  the  two  major  uses  of  a  game  or  simulation  are  teach¬ 
ing  or  research  alone. 
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PURPQSES  OF  THE  QUESTIONNAIRE 


1.  To  give  an  overall  quick  picture  of  the  purposes,  use,  benefits, 
and  costs  of  DOD  activities  in  gaining,  simulation,  and  the  use  of 
models. 

2.  To  provide  an  initial  description  and  classification  scheme  for 

a  professional  catalog  for  games,  simulations,  and  models  for  use 
in: 

o  Quick  professional  interchange  of  information  on 
the  characteristics  of  models. 

o  Aid  in  construction  and  estimation  of  model 
characteristics  for  new  model-builders. 

o  Aid  in  the  evaluation  of  previous  and  current 
activities. 
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INTRODUCTION :  TERMINOLOGY 


The  state  of  the  profession  is  such  that  no  clear 
agreement  on  fundamental  terms  exists  in  all  of 
the  various  activities  using  models,  simulations, 
or  games.  Acknowledging  this  fact,  we  advance 
the  following  definitions  to  provide  guides  to 
indicate  the  type  of  work  this  questionnaire  covers. 

Gaming:  A  gaming  exercise  employs  human  beings  acting  as  them¬ 
selves  or  playing  simulated  roles  in  an  environment 
which  is  either  actual  or  simulated.  The  players  may 
be  experimental  subjects  or  participants  in  an  exercise 
being  run  for  teaching,  operational,  training,  planning, 
or  other  purposes.  This  questionnaire  is  designed  to 
describe  games  constructed  for  these  various  purposes. 

War  Gaining:  One  of  the  major  applications  of  simulation  is  war  gaming. 

A  war  game  is  defined  by  the  Department  of  Defense  as  a 
simulation  of  a  military  operation  .involving  two  or  more 
opposing  forces  and  using  rules,  data,  and  procedures 
designed  to  depict  an  actual  or  assumed  real-life  situa¬ 
tion.  It  is  primarily  a  technique  used  to  study  problems 
of  military  planning  organization,  tactics,  and  strategy. 

A  war  game  can  be  conducted  to  cover  the  entire  spectrum 
of  war,  i.e.,  politico-military  crises,  general  war, 
or  limited  war.  The  game  may  be  based  on  hypothetical 
situations,  real-world  crises,  or  current  operational 
plans.  Some  games  are  designed  for  joint  operations  by 
two  or  more  military  services,  some  are  for  use  by  a  single 
service,  and  others  may  be  used  by  individual  Army  field 
commanders  or  even  by  division  or  battalion  commanders. 

The  level  of  command  at  which  the  game  is  to  be  played, 
of  course,  influences  the  type  of  units  to  be  represented 
and  the  scope  of  operations  to  be  conducted. 

There  are  three  types  of  war  games  in  common  use  today: 
the  training  game,  the  operational  game,  and  the  research 
game.  The  training  game  is  the  least  complex  and  is  de¬ 
signed  to  provide  the  participants  with  decisionmaking 
opportunities  similar  to  those  that  may  be  experienced 
in  combat.  The  operational  game  deals  with  current 
organizations,  equipment,  and  tactics.  It  is  more  complex 
than  the  training  game,  uses  inputs  that  are  based  on 
known  quantities,  and  is  used  to  test  operational  plans. 

The  research  game,  which  is  the  most  complex  of  the 
three  types  of  games,  requires  careful  preparation  to  achieve 
maximum  objectivity  and  usually  is  designed  to  study  tactical 
or  strategic  problems  in  a  future  time  frame. 


Simulation: 
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A  war  game  can  be  accomplished  manually,  can  be  computer- 
assisted,  or  can  be  wholly  computerized.  Manual  games 
are  played  using  symbols,  pins,  or  pieces  to  represent 
‘forces,  weapons,  and  targets  on  maps,  mapboards,  and 
terrain  models.  A  computer-assisted  game  is  a  manual 
game  using  computerized  models,  which  free  the  control 
group  from  many  repetitive  and  time-consuming  computations. 

Computer  games  are  based  on  predetermined  procedures. 

All  simulation  of  conflict  is  done  by  the  computer  in 
accordance  with  the  detailed  instructions  contained  in 
the  computer  program.  The  primary  advantage  of  computer 
gaming  is  that  the  same  situation  can  be  simulated  many 
times  under  differing  conditions,  to  observe  variability 
of  results.  A  computer  war  game  requires  the  use  of  a 
war  game  model  (i.e.  ,  computer  program),  which  contains 
all  the  rules,  procedures,  and  logic  required  to  conduct 
the  game. 

Simulation  involves  the  representation  of  a  system  or 
organism  by  another  system  or  model,  which  is  designed  to 
have  a  relevant  behavioral  similarity  with  the  origina 
system.  Games  utilize  a  simulated  environment  or  simu¬ 
lated  roles  for  the  players  or  both.  In  general,  all  games 
are  simulations.  However,  it  is  not  particularly  useful 
to  use  the  reverse  categorization.  In  other  words,  not 
all  simulations  are  usefully  regarded  as  games.  Computer 
simulations  stimulating  conflict  or  cooperation  (such  as 
completely  computerized  battle  models)  are  usefully  con¬ 
sidered  as  games.  Possibly,  so  are  some  logistic  or  resource 
allocation  models  where  the  single  (automated  or  live  player) 
team  may  be  regarded  as  struggling  against  a  statistical  or 
strategic  opponent  called  "Nature."  The  borderline  is  not  hard 
and  fast;  however,  we  would  expect  to  leave  out  a  straig  t 
industrial  production  scheduling  machine  simulation  from  our 
category,  for  example. 


Computer  simulation  is  an  analytical  technique  which 
involves  the  use  of  mathematical  and  logical  models  to 
represent  the  study  and  behavior  of  real-world  or  hypo¬ 
thetical  events,  processes,  or  systems  over  extended 
periods  of  time. 

Simulation  provides  the  means  for  gaining  experience  and 
for  making  and  correcting  errors  without  incurring  the 
costs  or  risks  of  actual  application.  It  offers  opportu¬ 
nities  to  test  theories  and  proposed  modifications  m 
systems  or  processes;  to  study  organizations  and  structures; 
to  probe  past,  present,  and  future  events;  and  to  utilize 
forces  that  are  difficult  or  impracticable  to  mobilize. 
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Simulation  therefore  is  of  value  both  as  an  educational 
device  and  as  a  means  of  discovering  improved  methods. 

Simulation  should  be  used  when  (1)  it  is  either  impos¬ 
sible  or  extremely  costly  to  observe  certain  processes  in 
the  real  world,  (2)  the  observed  system  is  too  complex 
to  be  described  by  a  set  of  mathematical  equations, 

(3)  no  straightforward  analytical  technique  exists  for 
solution  of  appropriate  mathematical  equations,  and 

(4)  it  is  either  impossible  or  very  costly  to  obtain 

data  for  the  more  complicated  mathematical  models  describing 
a  system. 

On  the  other  hand,  simulation  should  not  be  used  when 
(1)  simpler  techniques  exist,  (2)  data  are  inadequate, 

(3)  objectives  are  not  clear,  (4)  there  are  short-term 
deadlines,  or  (5)  the  problems  are  minor. 

Contract  Studies 

&  Analyses:  The  words  "studies’ and  analyses,”  as  used  in  this 
questionnaire,  refer  to  those  studies  and  analyses 
done  by  contract  or  by  grant  and  which  deal  with  the 
systematic  and  critical  examinations  of  various  subjects. 
Studies  and  analyses  often  require  advanced  analytical 
techniques  to  integrate  a  variety  of  factors  and  to 
evaluate  dat%.  Their  purpose  is  to  provide  greater 
understanding  of  alternative  organizations,  tactics, 
doctrines,  policies,  strategies,  procedures,  systems, 
and  programs. 

Model:  Conducting  a  computer  or  computer— assisted  war  game  or 

related  simulation  requires  the  use  of  a  computerized 
model.  As  used  in  this  context,  a  model  is  a  document 
or  program  containing  all  rules,  methodology,  techniques, 
procedures,  and  logic  required  to  simulate  or  approximate 
reality.  A  computerized  model  is  a  computer  program  or 
series  of  programs,  designed  to  simulate  the  logic  of 
actions  or  interactions  of  an  environment  or  a  context 
and  provide  the  results  to  player  personnel  for  subsequent 
analysis . 

MSG:  To  avoid  the  ambiguities  of  "local”  definitional  usage 

peculiarities,  we  have  elected  to  use  the  shorthand 
version  ”MSG"  to  stand  for  model/simulation/game  in  the 
remainder  of  the  questionnaire.  This,  we  hope,  will^ 
facilitate  present  communications  in  light  of  the  still 
unresolved,  serious  definitional  problems.  The  respondent 
is  encouraged  to  substitute  for  himself  whatever  local 
term  seems  appropriate  when  he  encounters  the  ”MSG”  label. 


cn  co  n  ^ 
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LABELS /ADMINISTRATION 
(For  Office  Use  Only) 


1-0  INTERNAL  ID  TAG:  ( _ _ _ ) 


2-0 

PUBLICATION 

ID 

NUMBER: 

( 

) 

2-1 

PUBLICATION 

ID 

NUMBER: 

c 

) 

2-2 

PUBLICATION 

ID 

NUMBER: 

( 

> 

2-3 

PUBLICATION 

ID 

NUMBER: 

( 

) 

2-4 

PUBLICATION 

ID 

NUMBER: 

c 

) 

-0 

ABSTRACTED: 

NO 

1-1 

ABSTRACTED: 

YES 

1  o 

•  T>r«mr>  «  . 

rU»Oir^AOiL.iJ  • 

onrmT  attti 

WVXX 

>-3 

ABSTRACTED : 

AUTHOR  ABSTRACT 
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INSTRUCTIONS 


In  filling  in  this  questionnaire,  as  many  questions  as  possible 
have  been  designed  to  be  answered  by  marking  one  or  more  relevant 
categories  in  each  question  (e.g.,  circling  or  marking  a  number). 

This  should  help  to  increase  the  speed  with  which  the  questions  can 
be  answered.  Furthermore,  it  is  easier  to  use  computer  assistance  to 
process  questions  of  this  type.  However,  in  many  instances  the  words 
used  for  the  categories  do  not  quite  reflect  the  properties  of  the 
game,  simulation,  or  study  being  characterized. 

The  respondents  are  requested  to  mark  the  most  relevant  categories^ 
then  to  make  any  written  commentary  modifying  the  answer  or  suggesting 
an  improvement  of  the  question. 

Notes  are  provided  on  pages  opposite  the  questions. 

Use  one  questionnaire  for  each  man-machine  game  or  simulation, 
machine  (or  analogue)  simulation,  or  study.  Parts  I,  II,  and  IV  of 
this  questionnaire  are  designed  to  be  answered  for  all  MSGs.  Part 
III  is  added  to  accommodate  the  particular  characteristics  of  man- 
machine  and  manual  games. 

Note  #1:  Please  note  the  amount  of  time  taken  to  fill  in  each 
questionnaire  and  return  to  Question  #1  to  record  this  information 
upon  completion  of  a  questionnaire. 

Note  ill:  To  the  right  of  each  question  there  is  a  three-interval 
confidence  scale.  If  you  are  certain  of  your  answer  mark  an  f,X" 
at  the  extreme  right  on  the  scale.  Please  mark  your  confidence 
level  for  all  questions  where  the  scale  is  indicated. 

Low  Middle  High 

1 _ I _ I _ A 

Confidence  Level 

On  a  scale  of  0-1  your  answers  will  be  interpreted  as  follows: 


Virtually  certain  .9-1 

High  .7  -  .9 

Middle  .3  -  .6 

Low  0  -  .  3 


! 
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If  you  wish  you  may  use  the  scale  in  either  of  two  ways ,  by  using 
an  X, 

1 - 1 - 1 - K - ■' 

or  by  putting  in  a  number. 

i _ i _ i  -8 _ i 

"Certain"  should  be  recorded  as  follows: 

1 - 1 - 1 - 1 


MODELS  AND  SIMULATIONS 

The  word  "model"  is  sometimes  used  to  refer  to  a  program  that 
might  be  called  a  general-purpose  subroutine  in  the  sense  that  it, 
together  with  other  models,  may  be  run  together  in  a  simulation.  Thus, 
many  simulations  may  be  specific  "one-shot"  configurations  of  models 
operated  only  to  answer  a  specific  question.  Sometimes  a  large-scale 
program  may  be  called  a  simulation  and  is  used  more  than  once  with 
different  inputs  or  even  with  models  or  subroutines  added.  If  you 
feel  that  an  important  technical  distinction  concerning  your  MSG  has 
not  been  made,  comment  accordingly  and  note  where  it  might  influence 
an  answer. 

BUILDERS  AND  USERS 

The  first  section  of  this  questionnaire  is  oriented  more  toward 
those  who  use  the  outputs  from  an  MSG  applied  to  a  particular  problem 
or  study  than  to  those  who  designed  or  built  the  MSG.  A  user  who  is 
reasonably  well  acquainted  with  the  technical  aspects  of  the  work  may 
be  able  to  answer  the  second  section  of  the  questionnaire.  However,  if 
it  appears  necessary  to  have  another  individual  or  group  answer  the 
second  part,  this  should  be  done  and  the  second  group  should  be 
identified. 
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Part  I:  Basic  Information  on  Purposes,  Use,  Benefits,  and  Costs 
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NOTE:  Coding  data ,  response  data,  and  other  information  not  in  the 
original  questionnaire  appear  in  italics. 


*1-0  QUESTIONNAIRE  TIME  ( _ _ _ ) 

[Those  Questions  marked  are  of  particular  importance  and 
therefore  extra  care  and  accuracy  are  called  for.] 


#2.  Simulation /Study /Mo del  and  Author  Name  Tag:  This  calls  for  the 
name  of  the  first  or  primary  effort,  list  name  of  the  agency  or 
firm(s)  that  built  the  game,  simulation  or  model  (MSG),  and  the  name 
of  the  two  or  three  major  authors  or  designers. 

The  agency  or  authority  is  the  official  group  ordering  the  work. 
The  sponsor  calls  for  the  name  of  the  officer  or  official  responsible 
for  ordering  the  work. 


*2-0  SIMULATION/STUDY/MODEL  NAME  ( _ _ _ _ _ 

DESIGNER  BUILDING  AGENCY  OR  FIRM  ( _ _ _ _ __ 

_ _) 

AUTHOR(S)  NAME(S)  ( - ) 

AGENCY  OR  AUTHORITY  (IES)  ( _ > 

( _ _ _ > 

SPONSOR(S)  OF  WORK:  NAME(S)( _ _ _ ) 


CATEG1,  3/10-11* 


3-0 

CATEGORY : 

3-1 

CATEGORY : 

3-2 

CATEGORY : 

3-3 

CATEGORY : 

3-4 

CATEGORY : 

3-5 

CATEGORY: 

NRfi . 

MODEL  . 

SIMULATION  . 

MAN-MACHINE  GAME  . 

MANUAL  GAME  . 

MATHEMATICAL  ANALYSIS  OR  STUDY 
OTHER  ( _ )  Specify  .. 


Response  (%) 

46.2 

34.8 

8.3 

2.3 
3.0 
4.5 
0.8 


^ This  information,  which  is  listed  for  various  questions  through¬ 
out  the  questionnaire,  indicates  the  code  name  of  the  variable  being 
studied  and  its  position  in  the  keypunch  deck.  Hence,  in  this  case, 
the  code  name  is  CATEG1,  occupying  card  3,  columns  10  and  11.  See  Ap¬ 
pendix  C  for  the  complete  codebook.  The  responses  are  percentages  o} 

132. 

^No  response. 
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# 4.  Respondent's  Role:.  [Describe  your  role  with  respect  to  this 
game  or  simulation,  e.g.,  participant/player/funder/user/designer/ 
implementor/caretaker ,  etc . ] 


RESR0L1,  2/ 12- 14 

*4-0  RESPONDENT'S  ROLE:  FUNDER  OR  SPONSOR  .  14.4 

4-1  RESPONDENT'S  ROLE:  USER .  21.8 

4-2  RESPONDENT'S  ROLE:  DESIGNER  OR  BUILDER  .  31.1 

4-3  RESPONDENT'S  ROLE:  PLAYER  .  0.0 

4-4  RESPONDENT'S  ROLE:  CARETAKER  .  13.6 

4-5  RESPONDENT'S  ROLE:  CONTROL  TEAM  OR  REFEREE  . .  1.5 

4-6  RESPONDENT’S  ROLE:  OTHER  (Specify)  .  7.6 


j [COMMENT :  Give  or  Attach  a  Brief  Job  Description] 


5-0  LENGTH  OF  TIME  YOU  HAVE  BEEN  ACQUAINTED  WITH 

THIS  WORK  ( _ _ _ _ ) 


6-0  IF  LESS  THAN  3  YEARS,  LIST  LAST  TWO  ASSIGNMENTS  ( 

_ )' 


*7-0 


RESPONDENT'S  INFORMATION:  NAME  ( _ _ _ ) 

RANK,  TITLE  AND  POSITION:  (  _ _ _ ) 

( _ - _ > 


SPECIAL  QUALIFICATIONS  AND/OR  EDUCATION  RELEVANT  TO  THIS 
TASK:  (Describe) 
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*8-0  MAJOR  STATED  GENERAL  PURPOSE  OF  YOUR  MODEL,  GAME,  OR  SIMULATION: 

[If  you  were  given  a  few  minutes  to  describe  the  project  to  a 
senior  official,  what  could  you  say?  (This  could  be  one  to  two 
double-spaced,  typed  pages,  or  if  you  have  a  reasonably  short 
written  summary,  simply  attach  a  copy.)] 


9-0  SPECIFIC  PURPOSES: 

[Name  two  specific  examples  of  questions  or  operational  problems 
this  MSG  has  been  used  to  answer.] 

[NOTE:  If  more  than  one  category  applies,  circle  more  than 

one  answer  in  your  reply . ] 

#10.  Purpose:  This  question  is  somewhat  redundant  with  #8  and  #9. 
However,  here  you  are  asked  to  conform  to  a  few— word  description  of  the 
work.  It  is  important  to  note  the  distinctions  made  in  #10-3  and  #10-4. 
Some  strategic  games  have  introduced  diplomatic  considerations  and 
international  bargaining.  These  would  fall  under  the  category  #10-3. 
Other  simulations  and  gaming  exercises  may  be  explicitly  concerned  with 
internal  economic  and  domestic  repercussions.  These  would  more  appro¬ 
priately  be  classified  under  #10-4.  It  is  of  course  possible  to  have 
exercises  that  may  be  classified  under  more  than  one  category,  such 
as  both  #10-3  and  #10-4. 


PURP1 , 

3/17-19 

* 

10-0 

PURPOSE: 

10-1 

PURPOSE 

10-2 

PURPOSE 

10-3 

PURPOSE 

10-4 

PURPOSE 

10-5 

PURPOSE 

10-6 

PURPOSE 

10-7 

PURPOSE 

TECHNICAL  EVALUATION  . 

DOCTRINAL  EVALUATION  . 

FORCE  STRUCTURE  EVALUATION  . 

MILITARY-DIPLOMAT IC  ANALY  SIS/ INTERNAT I0NAL 

RELATIONS  . . . . 

MILITARY-POLITICAL-ECONOMIC  ANALYSIS/DOMESTIC 

RELATIONS  . 

TRAINING/EDUCATION  . 

RESEARCH  METHODOLOGY  . 

OTHER  (Specify)  . 


44.  7 
18.2 
28.8 

0.8 

Q.O 

0.0 

1.5 

6.1 


NR  Low  Middle  High  Certain 


10CLt  3/20  ,  3.0  i  3.8  i  33. 6  ■  53.0  i _ 26^5 — , 

Confidence  Level 


#11.  Classification:  Often  simulations  or  models  per  se  are  unclas¬ 
sified.  However,  when  military  planning  factors  are  added  they  then 
become  secret  or  may  have  an  even  higher  classification.  This  means  that 
for  some  there  will  be  supportive  documents  that  fall  into  many  classi¬ 
fications.  #11-5  Proprietary.  We  include  business  classifications, 
such  as  simulations  run  internally  by  corporations  and  classified  as 
corporate  confidential;  furthermore,  proprietary  implies  that  the 
documents  are  made  available  only  by  the  proprietor  to  whomever^he 
sees  fit. 
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NA;  UNCLASSIFIED  .  72.0 

FOUO  .  6.8 

CONFIDENTIAL  .  12.9 

SECRET .  5.3 

TOP  SECRET .  0.0 

PROPRIETARY  .  3.0 

OTHER  (Specify)  .  0.0 


CLASWO ,  3/21 


11-0 

CLASSIFICATION 

11-1 

CLASSIFICATION 

11-2 

CLASSIFICATION 

11-3 

CLASSIFICATION 

11-4 

CLASSIFICATION 

11-5 

CLASSIFICATION 

11-6 

CLASSIFICATION 

CLASWI1,  3/22-23 

12-0 

CLASSIFICATION 

12-1 

CLASSIFICATION 

12-2 

CLASSIFICATION 

12-3 

CLASSIFICATION 

12-4 

CLASSIFICATION 

12-5 

CLASSIFICATION 

12-6 

CLASSIFICATION 

NR . 

(WITHOUT  INPUTS): 
(WITHOUT  INPUTS): 
(WITHOUT  INPUTS): 
(WITHOUT  INPUTS): 
(WITHOUT  INPUTS): 
(WITHOUT  INPUTS) : 
(WITHOUT  INPUTS): 

(INPUT  DATA,  OTHER)* 
(INPUT  DATA,  OTHER)* 
(INPUT  DATA,  OTHER)* 
(INPUT  DATA,  OTHER)* 
(INPUT  DATA,  OTHER)* 
(INPUT  DATA,  OTHER)* 
(INPUT  DATA,  OTHER) 


NA;  UNCLASSIFIED  ..  12.1 


FOUO .  0.0 

CONFIDENTIAL  .  6.8 

SECRET .  59.1 

TOP  SECRET  .  11.4 

PROPRIETARY .  0.0 


OTHER  (Specify)  ...  8.3 

. .  2.3 


* [DESCRIBE] 


#13.  Professional  Review.  What  professional  reviewing  procedures  are 
used  to  check  periodically  on  the  validity  of  this  MSG,  its  use  and 
its  inputs?  Is  there  any  formal  reviewing  group  external  to  both  the 
users  and  producers? 


PROREV,  3/24 

*13-0  PROFESSIONAL  REVIEW  DONE? 

BY  WHOM?  (Specify) 

RE  WAT,  3/25 

WHEN  LAST  DONE?  ( _ _ ) 


717  n  . . . 

.  40.2 

VT?C  . . 

.  49.2 

m?  . .  10.6 

Not  done . *  • « 

.  40.9 

<  6  months  . 

.  18.2 

6-12  months  . 

.  13.6 

13-24  months  . 

.  6.1 

25-36  months  . 

.  4.5 

>  36  months  . 

.  4.5 

NR . 

.  12.1 

#14.  MSG  Parent  or  Antecedents;  The  first  three  subcategories 
ask  for  information  on  a  formal  model  parent,  i.e.,  existing  model, 
simulation,  or  game  that  is  acknowledged  as  the  direct  parent  of  the 
model  categorized.  Even  for  original  models,  there  in  general  was  a  prior 
work  Or  specific  piece  of  literature  or  several  models  and  pieces  ol 
literature  whieh  provided  the  suggestion  or  stimulation  (positive  or 
negative)  for  the  construction  of  this  model. 
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MSGPAEj  3/26 

14-0  MSG  PARENT  OR  ANTECEDENTS,  DIRECT:  NONE .  37.9 

14-1  MSG  PARENT  OR  ANTECEDENTS,  DIRECT:  ONE . • .  87.9 

14-2  MSG  PARENT  OR  ANTECEDENTS,  DIRECT:  MORE  THAN  ONE  | 

(Specify)  .  20.5 

14- 3  MSG  MODEL  PARENT  OR  ANTECEDENTS,  INDIRECT,  NUMBER  (_ _ )  3.8 

NR  Low  Middle  High  Certain 

14CLj  3/27  ,  20^5  ,  4.5  ,  72.1  ,  43.9  , 18.  9_ — , 

Confidence  Level 

15- 0  NAME  DIRECT  PARENTS  ( _ _ _ ) 

( _ _ > 

16- 0  NAME  INDIRECT  ANTECEDENTS:  ( _ _ _ ) 


*#17.  Development  Initiation  Date:  Give  date  or  approximate  date  to 
nearest  month  or  year. 


*17-0  DEVELOPMENT  INITIATION  DATE:  ( 


#18 .  Use  Initiation  Date:  This  means  the  first  production  run. 

18-0  USE  INITIATION  DATE:  ( _ _ _ _ 


DEVTM1,  3/29 

ELAPSED  TIME  OF  DEVELOPMENT  (#18  - 


#17):  <  3  months .  7.6 

3-6  months .  19-  7 

7-12  months  .... .  21.2 

13-18  months  .  15.2 

19-24  months  .  8.3 

25-30  months .  8.3 

31-36  months  .  8.8 

37-42  months  .  8.8 

>  42  months .  8.0 

NR .  12.1 


#19.  MSG  Spinoff.  This  refers  to  models,  games  or  simulations  for 
which  this  MSG  was  a  parent.  We  also  count  the  same  MSG  used  elsewhere 
by  a  different  agency,  authority,  or  group  in  which  case  "same'  would 
be  entered  as  a  name  of  a  spinoff. 


MSGPRO j  3/28 

19-0  MSG  SPINOFF:  NONE  . . 

19-1  MSG  SPINOFF:  ONE  OR  MORE  (Specify  number)  1  . 

>  1 

NR 


60.6 

16.7 

17.4 

5.3 


Low  Middle  High 


l _ i - 1 - 

Confidence  Level 


20-0  NAMES  OF  ONE  OR  TWO  SPINOFFS:  ( 


) 


MODEL / S IMUL AT ION/ GAME  PRODUCTION  AND  PURPOSE 


#21.  Funding:  In  some  instances,  especially  when  individuals  do  a 
fair  amount  of  work  in  their  "spare  time,"  one  should  note  more  than 
one  funding  source.  Multiple  funding  sources  may  also  arise  when  a 
game  is  started  at  one  location  and  completed,  run,  or  used  at  another 
location.  NSF  would  be  classified  as  #21—1  and  #21—9. 


VH01,  Z/ZO-ZZ 


21-0 

FUNDING 

21-1 

FUNDING 

21-2 

FUNDING 

21-3 

FUNDING 

21-4 

FUNDING 

21-5 

FUNDING 

21-6 

FUNDING 

21-7 

FUNDING 

21-8 

FUNDING 

21-9 

FUNDING 

21-10 

FUNDING 

21-11 

FUNDING 

SOURCE: 

( 

)  . 

.. . .  0.0 

SOURCE: 

FOUNDATION  ( 

)  . 

_  0.8 

SOURCE: 

SOURCE: 

PRIVATE  (BUSINESS,  SELF, 
ARPA  ( 

MISC.)  ( 

)  2.Z 

)  .... 

•  •  •  •  0 .  o 

SOURCE : 

JCS  ( 

)  .... 

•  •  •  •  6 .  8 

SOURCE: 

USA  ( 

)  •  *  *  • 

....  42.4 

SOURCE: 

USAF  ( 

)  «... 

....  17.4 

SOURCE: 

USN  ( 

)  «  •  *  * 

....  25.0 

SOURCE: 

OTHER  DOD  ( 

)  •  •  •  • 

....  2.3 

SOURCE : 

OTHER  US  GOVERNMENT  ( _ 

_  ) 

....  0.8 

SOURCE: 

UNIVERSITY  ( 

)  .... 

•  •  •  •  2.3 

SOURCE: 

OTHER  ( 

)  •  •  i  • 

.  0.0 

#22.  MSG  Production:  #22-2  For  profit  includes  the  possibility  that 
a  game  is  built  by  a  for-profit  organization  but  not  necessarily  used 
as  a  product.  For  example,  it  might  be  used  for  research  or  for 
internal  training  purposes.  The  producer  may  be  a  for-profit  organiza¬ 
tion  building  for  the  DOD. 


WH0DID1 

!,  Z/Z4-Z5 

MSG  PRODUCTION: 

T5T3TT7ATr  TMT^T  VTnTT  AT .  . 

.  0.0 

22-0 

22-1 

22-2 

rKlVAi£ y  iru/ivii/u/Ui  ••••*»»•«. 

TTXTTXnTDCTTV  ............. 

.  5.Z 

MSG  PRODUCIION  : 

.  gf.  n  T1  A T\T 7CTT  /"YKT  • 

rnu  TJBHTTT  . . 

.  28.0 

MSG  PRODUCTION: 

A'Dvrc’n  PT?  Q  . . 

.  44.7 

22-3 

22-4 

MSG  PRODUCTION : 
MSG  PRODUCTION: 

NOT-FOR-PROFIT  . 

.  22.0 

#23.  MSG  Initiation:  #23-1  Model  builders /researchers  refers  to  a 
project  where  the  original  proposal  was  initiated  with  the  individua 
specifically  interested  in  researching  and  building  the  simulation. 

Much  of  MSG/research  work  falls  under  this  category.  The  researchers 
propose  the  construction  of  the  MSG  to  a  funding  or  a  sponsoring  agency. 
#23-2  MSG/users  may  initiate  a  proposal  for  construction.  For  examp  e, 
a  decision  may  be  made  to  run  a  model  at  an  institution  which  lias  a 
special  facility.  The  request  is  made  to  the  constructors  and  when  the 
simulation  is  ready,  those  who  requested  it  actually  participate  in  i  s 
operation. 
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#23—3  There  may  be  a  request  made  internally  to  an  organization  from 
a  management  group  for  the  construction  of  a  model  to  be  used  by  other 
individuals  in  the  organization.  In  other  words,  "a"  requests  "b"  to 
construct  a  model  to  be  used  by  nc.n  This  is  a  fairly  common  managerial 
structure.  #23— 4  An  agency  outside  of  an  organization  which .intends  to 
use  a  model  may  request  the  organization  to  build  it  for  the  agency. 

For  example,  the  local  governmental  group  may  decide  to  have  several 
sessions  with  an  urban  development  simulation.  They  may  request  private 
corporations  to  build  it  for  them  and  may  then  use  the  model  for  training, 
operational,  or  advocacy  purposes.  #23-5  An  outside  agency  may  request 
a  different  institution  to  both  build  and  use  a  simulation  for  their 
purposes.  In  this  case,  the  sponsor  is  really  only  interested  in  the 
results  and  not  in  the  specific  aspects  of  the  model.  For  example,  in 
certain  simulations  or  computerized  battle  models,  the  question  posed 
may  be  nWhat  are  the  characteristics  of  this  weapon  under  a  given  set 
of  circumstances?"  The  sponsor  may  approve  having  a  simulation  built  and 
constructed;  however,  the  sponsor's  technical  interest  per  se,  is  only 
limited  to  the  results  and  not  to  its  operation. 

INITIA 3  3/36 

*23-0  MSG  INITIATION: 

23-1  MSG  INITIATION: 

23-2  MSG  INITIATION: 

23-3  MSG  INITIATION: 

23-4  MSG  INITIATION: 

23- 5  MSG  INITIATION: 

#24.  Initiator  Purpose:  #24-1  The  differentiation  we  wish  to  make 
between  the  terms  "teaching"  and  "training"  is  that  training  is  more 
concerned  with  "how  to"  whereas  teaching  is  more  concerned  with  "why. 

In  many  instances  teaching  and  training  blend  imperceptibly  into  each 
other.  #24-2  Analysis.  This  meaning  is  to  be  distinguished  from 
#24-5  research/ theory  development.  Analysis  means  the  honest  grappling 
with  a  specific  question  or  set  of  questions  related  to  a  given  problem. 
The  distinction  is  best  made  between  operational  modeling,  where  a  purpose 
of  analysis  is  fairly  well  known,  and  academic  modeling,  where  research 
and  theory  development  are  more  the  norm. 

INPURP1 ,  3/37-39 

24- 0  INITIATOR  PURPOSE: 

24-1  INITIATOR  PURPOSE: 

24-2  INITIATOR  PURPOSE: 

24-3  INITIATOR  PURPOSE: 

24-4  INITIATOR  PURPOSE: 

24-5  INITIATOR  PURPOSE: 

24-6  INITIATOR  PURPOSE : 

24-7  INITIATOR  PURPOSE: 


24CL J  3/40 


NA;  UNKNOWN . . .  u-u 

TEACHING/ TRAINING . 

ANALYSIS /DIAGNOSTICS  .  81.8 

OPERATIONAL  .  10  *  6 

EXPERIMENTAL .  0.8 

RESEARCH /THEORY  DEVELOPMENT  . 5.3 

POPULARIZATION,  ADVOCACY  .  0.0 

OTHER  ( _ )  .  °-° 

NR  Low  Middle  High  Certain 

t  4.5  i  4.5  i  12.1  i  59.1  t  19.7__ | 

Confidence  Level 


NA;  UNKNOWN  . 

MODEL  BUILDERS /RESEARCHERS  .  18.2 

MODEL  USERS  INTERNAL  TO  ORGANIZATION  ...  30. 3 

NON-USER,  INTERNAL  TO  ORGANIZATION  .  10.6 

USER,  EXTERNAL  TO  ORGANIZATION .  23.5 

NON-USER,  EXTERNAL  TO  ORGANIZATION  .  12.9 
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#25.  Specificity  of  Purpose  of  Funding  Source:  The  specificity  of 
purpose  of  the  funding  source  will  in  general  depend  heavily  upon 
whether  or  not  the  funding  source  was  also  the  initiator  of  the  game 
project.  For  example,  a  proposal  may  be  made  to  ARP A  to  study  uncon¬ 
ventional  warfare.  They  will  sketch  out  certain  aspects  of  their 
proposal  and  more  or  less  what  they  are  going  to  do.  The  decision  to 
fund  this  will  be  based  upon  whether  or  not  the  group  appears  to  be 
competent  and  the  work  seems  to  be  "reasonable and  a  considerable 
amount  of  leeway  will  be  left  for  the  group’s  actual  work. 

Such  a  situation  would  fall  under  #25-2  moderately  specified  or  #25-3 
diffusely  specified.  On  the  other  hand,  there  may  be  a  specific  request 
from  a  government  agency  to  test  a  specific  piece  of  equipment  and  to 
use  a  study  or  simulation  to  report  on  the  quality  of  this  equipment. 

SPCPRP ;  3/41 

25-0  SPECIFICITY  OF  PURPOSE  OF  FUNDING  SOURCE:  NA;  UNKNOWN  ..  18.2 


25-1  SPECIFICITY  OF  PURPOSE  OF  FUNDING  SOURCE:  TIGHT  .  25.0 

25-2  SPECIFICITY  OF  PURPOSE  OF  FUNDING  SOURCE:  MODERATE . 40.9 

25-3  SPECIFICITY  OF  PURPOSE  OF  FUNDING  SOURCE:  DIFFUSE .  14.4 

NR .  1*0 


NR  Low  Middle  High  Certain 

25CLy  3/42  ,  6.1  i  6.1  i  29.5  \  42.4  t  15.9  \ 

Confidence  Level 

#26.  Best  Alternative  Procedures.  Imagine  that  the  objectives  of  the 
simulation  must  be  achieved  by  a  different  means.  It  might  appear  that 
to  check  none  and  then  to  check  something  else  would  be  mutually  in¬ 
consistent.  However,  if  the  MSG  is  used  for  more  than  one  purpose  this 
could  easily  arise.  For  example,  in  a  MSG  used  for  experimentation 
there  may  be  no  alternative  for  the  experiment,  yet  the  model  may  also 
be  used  for  teaching.  In  this  case,  lectures  or  case  studies  would  be 
reasonable  alternatives. 


ALTPR01 3  3/43-44 

26-0  BEST  ALTERNATIVE  PROCEDURE:  NA;  UNKNOWN  .  10.6 

26-1  BEST  ALTERNATIVE  PROCEDURE:  NONE  OR  VIRTUALLY  NONE . 22.0 

26-2  BEST  ALTERNATIVE  PROCEDURE:  LECTURES  .  0.0 

26-3  BEST  ALTERNATIVE  PROCEDURE:  CASE  STUDIES /HISTORY  .  11.4 

26-4  BEST  ALTERNATIVE  PROCEDURE:  ANALYSIS  .  43.2 

26-5  BEST  ALTERNATIVE  PROCEDURE:  EXPERIENCE .  6.8 

26-6  BEST  ALTERNATIVE  PROCEDURE:  GAMING .  0.3 


# 27 .  Major  Use  of  MSG:  Select  the  appropriate  categories  from  the 
list  below.  Also  provide  written  commentary  to  explain  what  you  mean 
by  the  categories  selected,  i.e.,  give  "for  instances." 
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USE1,  3/45-47 
27-0  MAJOR  USE  OF  MSG: 
27-1  MAJOR  USE  OF  MSG: 
27-2  MAJOR  USE  OF  MSG: 
27-3  MAJOR  USE  OF  MSG: 
27-4  MAJOR  USE  OF  MSG: 
27-5  MAJOR  USE  OF  MSG: 
27-6  MAJOR  USE  OF  MSG: 
27-7  MAJOR  USE  OF  MSG: 
NR . 


NA;  UNKNOWN  .  1-$ 

TEACHING/TRAINING .  2.0 

ANALYSIS /DIAGNOSTICS  .  79.5 

OPERATIONAL  .  s-8 

EXPERIMENTAL  .  0.8 

RESEARCH /THEORY  DEVELOPMENT  .  4.5 

POPULARIZATION,  ADVOCACY  .  0.0 

OTHER  (  )  .  0.0 

.  . .  0.8 


NR  Low  Middle  High  Certain 


27 CL }  3/48  >  3.0  i  3.0  I  34.4  |  55.3  i  21.2  i 

Confidence  Level 


*#28.  Analysis  Procedures:  Explain,  providing  for  instances,  the  type, 
amount,  and  rigor  of  analytic  procedures  used  on  the  output  of  this  MSG 

Low  Middle  High 


Confidence  Level 


#29.  Judged  Effectiveness  of  Best  Alternative  Procedure:  This  is 
for  the  main  purpose  of  the  MSG.  By  main  purpose  of  the  MSG  we  mean 
the  main  use  that  in  fact  has  been  made  of  it.  Initiator  purpose  had 
the  possibility  of  alternative  procedures  for  more  than  one  use.  In 
answering  this  question  we  restrict  ourselves  only  to  the  major  use. 

29-0  JUDGED  EFFECTIVENESS,  BEST  ALTERNATIVE  PROCEDURE:  ( _ ) 

Low  Middle  High 


Confidence  Level 


#30.  Number  of  Briefings:  Total  number  to  date  based  on  this  MSG. 


BRIEF }  3/49 

*30-0  NUMBER  OF  BRIEFINGS:  UNKNOWN  . 

30-1  NUMBER  OF  BRIEFINGS:  (Specify) 

0  . 

1-3 . 

4-6 . 

7-9 . . . 

10-12  . 

13-15  . 

16-18  . 

>  18 . 


42.4 

11.4 
11.4 

8.3 
0.8 

5.3 
1.5 
0.8 

18.2 


#31.  Level  of  Briefing:  By  level  of  briefing  we  refer  to  organization 
and  organizational  level  and  a  description  of  the  personnel  level  involved 
For  example,  number  and  rank  of  generals  or  senior  government  officials 
present. 


-96- 


*31-0  LEVEL  OF  BRIEFING:  [Describe] 


#32.  Purpose  of  Briefing:  Identify  and  state  purpose  of  two  most 
important  briefings,  i.e.,  what  decisions  depended  on  this  MSG? 


*32-0  PURPOSE  OF  BRIEFING: 


[Describe] 


#33.  Importance  of  MSG  to  Decision:  What  impact  did  MSG  have  on 
decision  just  noted?  Describe  specifically. 


*#34.  Measure  of  Benefits:  Specify  what  you  regard  as  a  reasonable 
measure  of  benefits  and  success  from  this  MSG. 
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MS  G  PRODUCTION  COSTS 


it 35.  Direct  Costs  to  Build:  Under  this  category  we  mean  actual 
expenditures  of  money  earmarked  for  the  purposes  of  developing  this 
particular  game,  formal  pay  for  working  time,  and  measures  attributed 
to  the  cost  of  the  game.  In  many  instances  where  there  is  a  great  amount 
of  informal  work  done,  the  direct  cost  for  the  development  of  the  game 
may  be  zero,  although  the  indirect  and  unallocated  costs  may  be  enormous. 
For  example,  the  UCLA  business  game  was  developed  heavily  by  the  use  of 
faculty  time  which  was  not  particularly  assigned  to  game  building.  Many 
university  games  have  this  property.  Many  games  built  in-house  to  an 
institution  without  direct  contractual  assistance  also  have  this  property. 
Thus,  the  question  should  be  construed  as  one  of  finding  out  when  direct 
monies  were  assigned  for  the  purpose  of  game  construction. 

CSTDIEj  3/50 

35-0  DIRECT  COSTS  TO  BUILD:  (Specify  to  nearest  $10,000) 


<  $49K .  19.7 

$50-$99K .  17.4 

$100-$249K .  13.6 

$250-$499K .  6.8 

$500-$749K .  6.1 

$750-$999K .  2.3 

$l-$2. 49  million  .  2.3 

NA-unknown-NR  . . 31.8 

NR  Low  Middle  High  Certain 


35CL ,  3/51  |  13. 6  |  10.6  i  30^3  i  35.6  i  I 

Confidence  Level 

it 36.  The  question  concerning  funding  for  development  and  building  is 
specifically  directed  at  the  accounting  question  of  what  money  has  been 
formally  assigned  to  the  effort  concerning  a  specific  model,  simulation, 
or  game.  Thus  it  will  almost  always  be  an  underestimation  of  cost. 


DIEFDS ,  3/54 

36-0  DIRECT  FUNDING  TO  BUILD:  NO  .  47.7 

36-1  DIRECT  FUNDING  TO  BUILD:  YES  .  43.9 

NE .  8.3 


37-0 


AMOUNT  OF  FUNDING  TO  BUILD: 

NR 

i  36.4 


37CL ,  3/56 


(Specify) 

Low  Middle  High 

,  8.3  i  19^7  i  2g.fi 

Confidence  Level 


Certain 

6.8  i 
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#38.  Total  Costs:  Direct,  Indirect,  Imputed,  and  Unimputed:  The 
ranges  are  purposely  kept  relatively  wide  open  because  of  the  extreme 
difficulties  in  accurately  judging  the  costs.  In  some  cases  a  variance 
of  100%  or  more  is  to  be  expected.  This  will  probably  be  a  rather 
surprisingly  large  number  if  one  is  honest  with  himself.  There  is  an 
obvious  political  problem  here.  Many  of  the  unimputed  costs  would  in  the 
course  of  time  have  an  opportunity  cost  of  zero.  Furthermore,  to  a  great 
extent  many  of  the  unimputed  costs  are  extra  hours  of  work  put  in  by 
oneself  and  not  paid  for.  Hence  they  do  not  necessarily  come  out  of  any¬ 
body’s  budget  or  funding.  They  might  be  called  the  n0ut  of  Hide  Costs." 


TOT C ST ^  3/57 

*38-0  TOTAL  COSTS  TO  BUILD:  DIRECT,  INDIRECT,  IMPUTED,  UNIMPUTED: 
(Specify  to  nearest  $10,000) 


<  $49K .  23.5 

$50-$99K . 14.4 

$100-$249K . . .  16.7 

$250-$499K . 12.9 

$500-$749K .  3.5 

$750-$999K .  3.5 

$l-$2.  49  million . * . . .  3.3 

$2.5-$5  million .  0.8 

NA-unknoim-NR . . .  18.9 


NR  Low  Middle  High  Certain 


S8CL 3  3/58  i  15.2  i  26.5  \  37.9  \  15.9  \  4.5  \ 

Confidence  Level 


MODEL /SIMULATION /GAME  OPERATING  COSTS  AND  OPERATIONS 


#39.  Under  costs  to  operate  we  include  professional  time,  support 
time,  set-up  time,  computer  time,  and  experimental  subject  or  participant 
time  together  with  overhead.  In  many  cases  many  of  these  items  are  pro¬ 
vided  free.  As  we  are  discussing  actual  expenditures,  the  estimates  we 
will  be  asking  for  will  be  gross  underestimates. 

There  is  possibly  more  variability  in  operating  costs  than  in  build¬ 
ing  costs  for  some  types  of  games.  The  variability  comes  in  the  way 
player  time  is  counted  as  a  cost  as  well  as  facilities’  use  and  operator’s 
time.  For  example,  with  many  business  games  played  at  universities  it 
would  appear  that  the  financial  costs  are  zero,  as  the  professorial  and 
student  time  is  not  regarded  as  a  direct  cost  and  in  some  instances 
computation  does  not  appear  as  a  directly  imputed  cost,  and  the  use  of 
facilities  which  would  otherwise  be  unoccupied  is  deemed  to  be  free. 

Any  attempt  to  replicate  that  game  in  an  environment  that  does  not 
have  these  features  may  be  extremely  costly. 

A  political  military  exercise  is  usually  run  as  an  individual  affair, 
and  we  should  consider  further  replications  even  though  they  might  be 
regarded  as  part  of  the  same  experiment  as  separate  items.  With  educa¬ 
tional  games  or  time-shared  experiments,  costs  are  calculated  based  on 
individual  game  or  subject  use.  For  simulation  the  cost  to  operate  is 
the  cost  to  explore  the  answer  to  a  typical  question.  This  is  relatively 
vague,  but  an  upper  bound  would  be  the  amount  of  exploration  that  might 
result  in  a  small  separate  publication. 
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CSTOPSj  3/59 

*39-0  ANNUAL  COST  TO  OPERATE,  GROSS:  (Comment,  if  necessary.) 

Year  Cost 

3-9 _  _ 

19 _  _ 

19  Specify  for  last  five  years. 

19~  _ 

19 _  _ 

<  $9K  .. 

$10-$24K 
$25-$49K 
$50-$74K 
$75-$99K 

>  $10 OK 

NR . 

NR  Low  Middle  High  Certain 

Z9CL,  3/60  i  25.  8  ,  22.7  i  33j_3  {  12j_9  i  5.5  i 

Confidence  Level 

#40.  Cost  to  Operate,  Single  Use:  What  does  it  cost  for  a  single  use 
of  the  MSG,  assuming  only  minor  or  no  variations  in  input  values?  For 
example,  name  such  a  use  and  date  it  approximately.  By  use,  we  mean 
for  a  single  study  effort.  (This  of  course  may  vary,  but  give  an 
average  estimate.) 

CSTRUNj  3/61 

*40-0  COST  TO  OPERATE,  SINGLE  USE: 

<  $9K . 

$10-$24K  .... . 

$25-$49K  . 

$50-$74K  . 

$75-$99K  . 

>  $10 OK  . 

NR . 

#41.  Annual  Update  Costs:  If  the  model  is  under  continuous  development, 
what  are  the  annual  costs  of  these  activities,  over  and  above  normal 
operating  costs? 

CSTUT,  3/62 

*41-0  ANNUAL  UPDATE  COSTS,  TOTAL  DOLLARS:  ( 

<  $9K . 

$10-$24K  . 

$25-$49K  . 

$50-$74K  . 

$75-$99K  . 

>  $100K  . 

NR . 


_ ) 

27.3 

6.8 

13.6 

2.3 

1.5 

2.3 

46.2 


49.2 

9.8 
2.3 
0.8 
2.3 

3.8 
31.8 


23.5 
16.7 

7.6 

6.8 

2.3 

10.6 
32.  6 


41-1  ANNUAL  UPDATE  COSTS,  PROFESSIONAL  MAN-YEARS:  ( _ ) 

41-2  ANNUAL  UPDATE  COSTS,  PROGRAMMER  MAN-YEARS : (  _ ) 

NR  Low  Middle  High  Certain 

41CL,  3/63  i  27.5  ,  12.9  i  27.3  i  2^0  i  7.6  \ 

Confidence  Level 

#42 .  Operational  Life  Span  (to  date):  By  operational  life  span  we  mean 
the  period  starting  after  development  is  complete,  beginning  with  the  first 
operational  use  of  the  model  to  the  period  when  it  and  its  analysis  or 
post— run  exposition  are  set  aside,  having  served  their  purpose. 

If  a  model,  simulation, or  game  that  has  been  in  more  or  less  continuous 
use  is  still  in  operation,  both  that  length  of  time  of  operation  and  the 
fact  that  it  is  still  operational  should  be  indicated. 

0PLIF1 ,  3/64 

*42-0  OPERATIONAL  LIFE  SPAN  (TO  DATE) :  (Specify) 

<  3  months  . 

3-6  months  . 

7-12  months  . 

13-18  months  . 

19-24  months  . 

25-30  months . . . . 

31-36  months  . 

'  37-42  months •  . . . . . . 

>42  months  . 

NE . 

NR  Low  Middle  High  Certain 

42CLj  3/65  ,  12.1  i  3.8  i  15.2  i  3BJ6  l  2j\3  | 

Confidence  Level 

#43.  Still  Active?:  By  this  we  mean,  "Is  the  MSG  in  its  original  or 
mildly  modified  form  still  actively  being  used  for  production  runs?" 

We  contrast  this  with  a  serious  revision  that  has  resulted  in  either 
a  new  name  or  an  appellation  such  as  Mark  II,  or  Mod  III,  and  so  on. 

ACTIVE j  3/66 

*43-0  STILL  ACTIVE?:  NO 
43-1  STILL  ACTIVE?:  YES 

NR 

NR  Low  Middle  High  Certain 

43CL ,  3/67  i  9.8  i  0.8  i  6.1  i  SOJj  i  IfLg—J 

Confidence  Level 

*44-0  MODEL  USERS : 

[Specify  all  agencies  who  directly  use  the  MSG  outputs  by  full 
name  (not  military  abbreviations);  indicate  the  prime  user.] 


27.3 

65.2 

7.6 


9.1 

7.6 

8.3 

6.8 

14.4 
3.0 
8.3 
0.-8 

23.5 

18.2 
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* 

#45.  Operational  Use:  Annual  Frequency:  By  this  we  are  trying  to 
find  out  how  many  times  this  MSG  is  actually  exercised  or  run  annually. 
Specify  for  the  last  five  years. 

Year  Number  of  Times 

19 _  _ 

19 _  _ 

19 _  _ 

19 _  _ 

19  _ 


0PFRQ1 ,  3/68 

Not  operational .  16.7 

I- 5 .  23.5 

6-10  .  6.8 

II- 15 .  4.5 

16-20 .  1.5 

21-25  .  5.0 

26-50  .  7.6 

51-100 .  9.1 

>  100 .  15.9 

NR .  H.4 

EXPOSE,  3/69 

46-0  EXPERIMENTAL  USE:  NO  .  76.5 

46-1  EXPERIMENTAL  USE:  YES  .  17.4 

NR .  6.1 


#47.  Experimental  Example:  If  46-1,  i.e.,  "yes,"  give  a  for  instance. 
[Des cribe] 

#48.  Experimental  Purpose,  Initial:  Was  this  MSG  designed  originally 
for  experimental  use? 


EXPRP,  3/70 

48-0  EXPERIMENTAL  PURPOSE,  INITIAL:  NO  . .  86.4 

48- 1  EXPERIMENTAL  PURPOSE,  INITIAL:  YES  .  9.8 

NR .  5.8 

EDUSE,  3/71 

49- 0  EDUCATIONAL  USE:  NO  .  91.7 

49-1  EDUCATIONAL  USE:  YES  .  5.3 

NR .  5.0 


#50.  Educational  Example:  If  49-1,  i.e.,  "yes,"  give  a  for  instance. 
[Describe]  _ _ 

#51.  Educational  Purpose,  Initial:  Was  this  MSG  designed  originally 
for  educational  use?  — . - 


EDPR0,  3/72 

51-0  EDUCATIONAL  PURPOSE,  INITIAL:  NO .  91.7 

51-1  EDUCATIONAL  PURPOSE,  INITIAL:  YES  .  1.5 

NR .  8.8 
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#52.  Transferability  of  MSG  Use:  #52-0  is  a  model  not  intended  for 
transfer.  An  example  of  such  might  be  a  classified  simulation  run  to 
test  for  some  particular  parameter  value  and  dispensed  with  after  the 
runs;  or  a  study  or  simulation  may  be  extremely  classified  with  only 
one  user  in  mind.  #52-1  This  would  be  something  like  chess  or  Go  where 
once  the  rules  have  been  transmitted,  one  could  take  a  piece  of  paper, 
draw  the  board,  obtain  some  stones,  and  play*  In  general,  a  game  of 
this  variety  can  be  transferred  immediately  at  little  or  no  cost.  #52-2 
An  example  of  a  game  in  this  category  would  be  a  game  that  is  not  overly 
complex,  is  extremely  well  documented  and  produced,  and  is  generally 
available.  The  games  Simsoc  or  Whifn* proof  or  Summit  or  Democracy  would 
all  fit  into  this  category. 

52-3  Middling  would  cover  simulation  games  such  as  the  UCLA  business 
game  where  the  size  of  the  program  is  not  enormous  and  computer  require¬ 
ments  are  such  that,  at  least  in  the  United  States,  many  institutions 
would  have  the  facilities.  Furthermore,  the  documentation  is  reasonably 
good.  To  get  it  operating,  such  a  game  would  require  one  or  two  months, 
taking  into  account  telephone  calls,  time  delays  in  the  mails,  etc. 

52-5  Simulations  that  are  extremely  difficult  to  travel  are  ones  that 
depend  upon  specific  facilities  and  crews  of  experienced  individuals  or 
that  are  enormous  in  computational  size.  For  that  matter  they  may 
not  be  computer  games,  but  have  become  so  large  that  they  should  be 
regarded  more  as  institutions  rather  than  games.  Examples  of  such  are 
METRO,  The  Rand  Logistic  Lab  experiments,  TEMPER,  and  the  Carnegie  Tech 
game.  The  reasons  why  these  cannot  travel  are  different;  however,  in 
each  case  the  amount  of  work  required  to  transfer  the  operation 
is  enormous. 

TRANSUj  3/73 


52-0 

CO  1 

TRANSFERABILITY 
T'RANQ'FF'PARTT  TTY 

OF  MSG 
OF  MSG 
OF  MSG 
OF  MSG 
OF  MSG 

ATT 

USE: 

USE: 

USE: 
USE: 
USE: 
ttQF  • 

NOT  INTENDED  FOR  TRANSFER  . . 
GENERALLY  . 

10.6 

18.2 

JZ  1 

52-2 

52-3 

CO_A 

X  O  r  Hi  rvrtJj  X  XiX  X  X 

TRANSFERABILITY 

TRANSFERABILITY 

TRANSFERABILITY 

TO  Aiac  A'RTT  T  TV 

MODERATE  DIFFICULTY  . 

MIDDLING  DIFFICULTY  . 

nTFVTGTTT.T  . 

17.4 
26.  5 
18.2 

c  o  c 

T7YTPFMF  TiT  FFT  f.TTT.TY  . 

7.  6 

1KAIN  dX£jKAI>XL»1  X I 

NR . 

ur  m  Kj 

UlML*  • 

LAI ILul (Ulf  liwWUA  a  •  •  •  •  *  •  •  •  ■ 

1.5 

[Describe  whether  transferability  just  indicated  pertains  to  any  of 
the  following  situations:  (1)  use  by  another  person  or  organization  at 
a  new  site,  (2)  use  by  the  same  developer/designer /builder  at  a  new  site 
or  (3)  use  by  another  person  or  organization  at  site  where  MSG  presently 

operates. ] 


#53.  Transferability:  Costs  to  Operate:  Using  information  provided 
in  #52,  please  estimate  how  much  additional  amount,  with  respect  to 
normal  operating  costs,  would  be  required  to  transfer  and  then  operate 
this  MSG. 
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TRANSC,  3/74 


53-0 

TRANSFER 

COST 

TO 

OPERATE: 

53-1 

TRANSFER 

COST 

TO 

OPERATE: 

53-2 

TRANSFER 

COST 

TO 

OPERATE: 

53-3 

TRANSFER 

COST 

TO 

OPERATE: 

53-4 

TRANSFER 

COST 

TO 

OPERATE: 

53-5 

TRANSFER 

COST 

TO 

OPERATE: 

53-6 

TRANSFER 

COST 

TO 

OPERATE: 

NR 

53CL,  3/75  ,  13 


NA;  UNKNOWN .  45.5 

COST  .  40.6 

COST-COST+10%  .  42.9 

C0ST+10%-C0ST+25%  .  48.2 

COST+25%-COST+50%  .  3.0 

COST+50%-COST+100%  .  6.8 

>  COST+100%  .  3>.0 


Low  Middle  High  Certain 

6  [  16.7  i  26.6  i  30.6  i  10. 6  \ 

Confidence  Level 


#54.  Obsolescence:  Comment  on  the  speed  at  which  you  expect  this 
MSG  to  become  obsolescent,  indicating  the  reasons  why.  We  are  not 
referring  to  the  need  for  reprogramming  for  new  hardware  or  for  minor 
modifications  but  to  the  state  where  it  is  no  longer  sufficiently 
relevant  that  either  a  major  modification  has  to  be  made  or  it  is 
completely  abandoned.  For  example,  a  special  simulation  may  be  built 
and  run  once  for  a  specific  purpose.  A  simple  model  may  be  used  for 
many  years ,  as  long  as  the  type  of  damage  calculation  it  performs 
is  relevant. 

#55.  Related  MSG:  Does  there  exist  an  MSG  that  might  be  regarded  as 
serving  approximately  the  same  purpose  as  yours? 

DUPMSG,  3/76 

55-0  RELATED  MSG:  NO 
55-1  RELATED  MSG:  YES 
NR . 


52.3 

43.9 

3.8 


#56.  Duplication  of  Use:  If  55-1,  i.e.  "yes,"  name  the  MSG(S) 
and  state  why  or  why  not  one  MSG  could  serve  the  purposes  of  all. 
If  your  answer  is  "no,"  i.e.,  55-0,  state  why  you  believe  there  is 
no  MSG  similar  to  yours. 


#57.  Clearing  House:  Is  there  enough  communication  in  DOD  among 
different  gaming  and  simulation  studies  or  would  a  clearing  house 
or  central  agency  be  of  use? 


CLEAR,  3/77 


57-0 

CLEARING  HOUSE: 

HIGHLY  USEFUL 

57-1 

CLEARING  HOUSE: 

USEFUL  .  . . 

57-2 

CLEARING  HOUSE: 

SAME  AS  IS  ... 

'57-3 

CLEARING  HOUSE: 

HARMFUL  . 

57-4 

CLEARING  HOUSE: 

HIGHLY  HARMFUL 

NR . 

44.4 
43.2 

23.5 

8.3 

5.3 
5.3 


Comments  on  Question  #57  are  required: 
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#58.  Standardization:  After  techniques  and  studies  have  been  in 
existence  for  some  time,  standardization  and  exchange  of  common 
routines  is  extremely  useful.  However,  sometimes  premature  attempts 
to  standardize  do  more  harm  than  good.  In  particular,  redundancy  to 
an  outsider  may  not  be  redundancy  to  those  doing  a  study.  What  is 
your  belief  in  the  advisability  of  increasing  DOD  gaming  and  simulation 
activity  for  standardization? 

STNRD1 ,  3/78 


58-0  STANDARDIZATION:  HIGHLY  USEFUL  .  6.8 

58-1  STANDARDIZATION:  USEFUL  .  24.2 

58-2  STANDARDIZATION:  SAME  AS  IS  .  22.7 

58-3  STANDARDIZATION:  HARMFUL  .  31.1 

58-4  STANDARDIZATION:  HIGHLY  HARMFUL  .  10.6 

NR .  4.5 

Comments  on  Question  #58  are  required: 

Low  Middle  High 


I _ I _ I _ I 

Confidence  Level 

#59.  Regional  Centers:  Models,  simulations,  and  games  are  operated 
and  built  at  many  locations  using  many  languages  and  different  staffs 
and  equipment.  Would,  for  example,  3-5  appropriately  cleared  regional 
centers  for  most  of  these  activities  be  a  more  or  less  effective  way 
of  supporting  them?  Discuss. 


REGCENj  3/79 

HIGHLY  USEFUL  .  0.8 

USEFUL  .  12.9 

SAME .  8.3 

HARMFUL .  46.2 

HIGHLY  HARMFUL .  14. 4 

NR .  17.4 


#60.  External  Review  Board:  Would  you  comment  on  reasons  both  pro 
and  con  having  an  external  review  board  consider  this  and  other  MSG's 
built/operated/used  by  your  organization  or  activity?  Discuss. 


EXREV,  3/80 

HIGHLY  USEFUL  .  0.8 

USEFUL .  14.4 

SAME .  15.9 

HARMFUL . 59.4 

HIGHLY  HARMFUL .  17.4 

NR .  12.1 


#61.  External  Review  Board,  Specification:  Would  you  characterize 
a  "good"  external  review  board  as  to  composition  and  function?  Discuss. 


Part  II:  Model/Simulation/Game  Characterization  and  Description 


NOTE:  If  Part  II  not  filled  out  by  Office  of  Prime  Responsibility 

and/or  user,  please  indicate  who  filled  out  Part  II. 

Name  _ _ _ _ 

Organization  _ _  _ 

Relationship  to 
Prime  Office  of 

Responsibility  _ _ 
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#62.  Scenario  Type:  #62-0  means  that  the  inputs,  outputs,  and  inter¬ 
pretation  of  the  outputs  are  all  numerical  (example:  number  of  targets 
destroyed).  #62-1  means  that  a  verbal  description  of  the  scenario  must 
be  available  prior  to  use  (example:  a  man-machine  or  free-form_game)  . 

#62-2  means  that  the  interpretation  of  an  output  is  qualitatively 
modified  or  interpreted  prior  to  being  used  (example:  a  written 
assessment  of  qualitative  aspects  of  target  damage  may  accompany  a 
computer  output) • 

SCENTPj  4/7 

62-0  SCENARIO  TYPE: 

62-1  SCENARIO  TYPE: 

62-2  SCENARIO  TYPE: 

NR . 

#63.  Scenario  Description:  Rich  "realistic"  may  be  used  to  refer  to 
a  scenario  which  is  both  rich  in  detail  and  purports  to  be  a  realistic 
description  of  some  phenomenon.  For  example,  some  tactical  games  may 
go  to  great  lengths  to  have  a  realistic  description  of  weather  conditions, 
troop  morale  conditions,  terrain  conditions,  details  on  buildings,  and 
so  forth.  The  measure  of  the  description  of  environment  should  be 
relative  to  the  real-world  phenomenon  being  modeled.  For  example,  a 
business  game  might  have  fewer  details  in  it  than  a  diplomatic  game 
but  be  a  richer  model  in  relation  to  actuality  than  the  diplomatic 
game.  Furthermore,  some  games  may  have  underneath  them  a  mathematical  .  ^ 

model  which  is  not  necessarily  apparent  to  the  users.  The  word  imaginative 
can  be  used  to  refer  to  nonfactual  modeling  where  a  scenario  may  contain 
counterfactual  or  futuristic  features.  These  aspects  may  be  mixed  in 
with  other  environmental  categorizations. 

*63-0  SCENARIO  DESCRIPTION:  [Describe] 


ONLY  NUMERICAL  .  53.8 

VERBAL  DESCRIPTION  NEEDED  FOR  USE  .  20.5 


VERBAL  DESCRIPTION  NEEDED  FOR  ANALYSIS  ..  22.7 


#64.  Mathematical  Sophistication  of  MSG:  #64-0,  None  equals  less 
than  high  school  math  needed  to  interpret  output  or  to  participate  as 
a  player  if  it  is  a  game.  #64—1,  Slight  equals  high  school  math. 
#64—2,  Moderate  equals  needs  college  level  math  (BA)  or  engineering 
degree.  #64-3,  High  equals  requires  an  advanced  degree  to  interpret 
the  output. 


MATES,  4/8 

64-0  MATHEMATICAL  SOPHISTICATION 
64-1  MATHEMATICAL  SOPHISTICATION 
64-2  MATHEMATICAL  SOPHISTICATION 
64-3  MATHEMATICAL  SOPHISTICATION 
NE . 


OF  MSG:  NONE .  0.8 

OF  MSG:  SLIGHT  .  29.5 

OF  MSG:  MODERATE .  59.1 

OF  MSG:  HIGH  .  8.3 

.  2.3 

Low  Middle  High 


i _ i _ 

Confidence  Level 


L 


x 
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#65.  Timing  of  Moves:  Event  timing  implies  that  moves  depend  upon 
a  specific  event  having  occurred.  Fixed  clock  timing  implies  that  there 
is  a  certain  increment  of  time  upon  which  model  activities  are  based. 
There  are  some  models  that  are  both  event  and  fixed  clock,  in  the  sense 
that  generally  the  clock  moves  forward  at  regular  periods,  e.g.,  descrip¬ 
tions  of  gross  national  product  in  an  international  model.  However, 
simultaneously  there  may  be  moves  that  depend  on  specific  events,  such 
as  conditional  checking  for  threshold  effects,  time  in  queues,  etc. 


MOVES ,  4/9 

65-0  TIMING  OF  MOVES:  NA;  UNKNOWN 

65-1  TIMING  OF  MOVES:  EVENT  . 

65-2  TIMING  OF  MOVES:  FIXED  CLOCK 

COMBINATION  65-1  and  65-2  - 

NR . 

#66.  Model  Time  to  Real  Time  Ratio: 


.  21.2 

.  59.4 

.  22.0 

.  15.2 

.  2.3 

In  describing  the  ratio  of  model 


time  to  real  time  one  has  the  problem  of  distinguishing  between  the 
period  assigned  the  real  time  and  the  amount  of  that  time  which  would 
have  been  used  for  the  decisionmaking.  For  example,  in  a  model  in  which 
the  real  time  is  meant  to  be  quarters,  the  price  decision  in  a  market  may 
only  take  a  week  or  two  to  make.  In  the  exercise,  twenty  minutes  may  be 
allotted  for  the  decisionmaking.  We  now  have  the  problem  of  deciding 
whether  to  scale  the  twenty  minutes  against  the  one  week  or  the  three 
months.  We  suggest  scaling  against  the  allotted  real  time,  i.e.,  the  three 


months . 


MT/RT,  4/10 

66-0  MODEL  TIME  TO  REAL  TIME  RATIO:  (Specify) 

NA-UNKNOWN  . 

HIGHLY  COMPRESSED  . 

COMPRESSED  . 

REAL  TIME  . 

EXPANDED  . . 

HIGHLY  EXPANDED  . 

NR  * . . . 


25.0 

14.4 

34.1 

12.1 
7.6 
1.5 
5.3 


NR  Low  Middle  High  Certain 


6 6 CL,  4/11  i  11.4  i  14.4  i  25. 8  ,  37_A  \  TP4 — | 

Confidence  Level 


#67.  Time  Represented:  In  some  instances  where  the  simulator  is 
not  necessarily  meant  to  represent  any  specific  structure  the  time 
i  period  represented  might  be  interpreted  as  the  present,  but  it  might  also 
be  better  to  describe  it  as  unspecified.  In  cases  of  doubt,  it  is  best 
to  note  the  model  in  both  categories. 


TIME , 

4/12 

67-0 

TIME  REPRESENTED: 

PAST  . 

67-1 

TIME  REPRESENTED: 

PRESENT  . 

67-2 

TIME  REPRESENTED: 

FUTURE  . 

67-3 

TIME  REPRESENTED: 

UNSPECIFIED  , 

67-4 

TIME  REPRESENTED: 

COMBINATIONS  . 

NOT  RELEVANT 

NR 


0.0 

9.8 

18.9 
33.3 

15.9 

18.9 
3.0 
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#68.  Level  of  Resolution,  Model  Time:  This  is  the  smallest  time  unit 
recognized  by  the  game. 


LRTIME ,  4/13 

68-0  LEVEL  OF  RESOLUTION,  MODEL  TIME:  NA;  UNKNOWN  .  18.9 

68-1  LEVEL  OF  RESOLUTION,  MODEL  TIME:  SECONDS  .  38.6 

68-2  LEVEL  OF  RESOLUTION,  MODEL  TIME:  MINUTES  .  21.2 

68-3  LEVEL  OF  RESOLUTION,  MODEL  TIME:  HOURS  .  10.6 

68-4  LEVEL  OF  RESOLUTION,  MODEL  TIME:  DAYS, .  6.8 

68-5  LEVEL  OF  RESOLUTION,  MODEL  TIME:  WEEKS  .  0.8 

68-6  LEVEL  OF  RESOLUTION,  MODEL  TIME:  QUARTERS  .  0.0 

68-7  LEVEL  OF  RESOLUTION,  MODEL  TIME:  YEARS  .  0.0 

68-8  LEVEL  OF  RESOLUTION,  MODEL  TIME:  >  YEARS  .  0.0 

NR .  2.0 

Low  Middle  High 

i _ i - - — i - 1 


Confidence  Level 

#69.  Level  of  Resolution,  Space:  In  military  games,  the  spatial  level 
of  resolution  is  frequently  important j  in  most  business  games  spatial 
level  of  resolution  is  at  best  crude.  #69-4  refers  to  the  situation  where 
detail  may  be  supplied  for  specific  locations,  but  no  detail  is  given 
between  them:  for  example,  details  of  the  terrain  around  enemy  airports, 
but  no  details  for  terrain  between  them. 


LRSPCEj  4/14 

69-0  LEVEL  OF  RESOLUTION,  SPACE:  NA .  9.8 

69-1  LEVEL  OF  RESOLUTION,  SPACE:  SMALL  AREA  (METERS)  .  34.8 

69-2  LEVEL  OF  RESOLUTION,  SPACE:  MODERATE  (KILOMETERS)  .  26.5 

69-3  LEVEL  OF  RESOLUTION,  SPACE:  LARGE  (THEATER/CONTINENT)  ..  12.9 

69-4  LEVEL  OF  RESOLUTION,  SPACE:  VARIED  .  12.6 

NR .  2- 1 


NR  Low  Middle  High  Certain 

69CL ,  4/15  ■  fi.3  i  2.3  i  l^d  \  \  Mil  \ 

Confidence  Level 

#70.  Level  of  Resolution,  Sides:  In  some  models  for  some  purposes  there 
is  no  need  to  resolve  the  nature  of  individual  teams.  Gross  performances 
of  the  interaction,  as  a  whole  are  being  considered  regardless  of  team  size. 
For  other  purposes  the  same  model  may  be  used  with  considerable  attention 
paid  to  the  team  structure. 

Furthermore,  a  distinction  between  structured  and  unstructured  groups 
must  be  made.  In  6ome  instances,  e.g.,  when  studies  of  the  emergence  of 
leadership  are  being  conducted,  it  is  important  that  no  structure  be 
placed  on  the  teams.  In  other  cases  the  teams  may  be  given  a  structure 
such  as  that  of  a  corporation  or  a  military  command. 

#70-7.  As  platoon,  division,  air  force,  etc.,  vary  in  size  between  the 
services ,  name  the  generic  term  for  the  unit. 


-109- 


LRSIDE3  4/16 

70-0  LEVEL  OF  RESOLUTION,  SIDES:  NA  .  20.3 

70-1  LEVEL  OF  RESOLUTION,  SIDES:  INDIVIDUALS  .  23.5 

70-2  LEVEL  OF  RESOLUTION,  SIDES:  SMALL  GROUPS  (STRUCTURED)  ..  19.7 

70-3  LEVEL  OF  RESOLUTION,  SIDES:  ORGANIZATIONS,  SMALL  (<^ 100)  13.6 

70-4  LEVEL  OF  RESOLUTION,  SIDES:  ORGS.,  LARGE  (>  1000)  .  4.5 

70-5  LEVEL  OF  RESOLUTION,  SIDES:  ORGS.,  VERY  LARGE  .  2.0 

70-6  LEVEL  OF  RESOLUTION,  SIDES:  SMALL  GROUPS,  UNSTRUCTURED  .  0.0 

70-7  LEVEL  OF  RESOLUTION,  SIDES:  NAME  UNIT  (  )  .  2.0 

NR .  2 'l 

_  Low  Middle  High 

Comment : 

I _ » - 1 - 1 

Confidence  Level 


#71.  Level  of  Resolution,  Military  Action:  The  categories  here  are 
arranged  in  order  of  progressive  generality,  thus  #71-5  includes  war  as 
a  part  of  ongoing  diplomacy. 

LRMIL,  4/17 

71-0  LEVEL  OF  RESOLUTION,  MILITARY  ACTION:  NA;  UNKNOWN  .  19.7 

71-1  LEVEL  OF  RESOLUTION,  MILITARY  ACTION:  ENGAGEMENT  .  44.7 


71-2  LEVEL  OF  RESOLUTION,  MILITARY  ACTION:  BATTLE  .  10.6 

71-3  LEVEL  OF  RESOLUTION,  MILITARY  ACTION:  CAMPAIGN  .  11.4 

71-4  LEVEL  OF  RESOLUTION,  MILITARY  ACTION:  WAR  .  12.6 


71-5  LEVEL  OF  RESOLUTION,  MILITARY  ACTION:  DIPLOMATIC  .  0.0 

Comment:  [Note  specific  details  of  this  MSG. ] 

NR  Low  Middle  High  Certain 

71CL,  4/18  i  6,g  i  3.6  i  12^6  |  45.5  i  27.3 | 

Confidence  Level 


ini.  Random  Events:  It  is  possible  to  use  a  model  occasionally  with 
random  events  and  on  other  occasions  without  random  events.  In  this 
case  both  categories  should  be  checked.  One  should  not  regard  this 
question  as  a  binary  choice;  both  are  possible. 


RNDM,  4/19 

72-0  RANDOM  EVENTS :  NO 

72-1  RANDOM  EVENTS:  YES 
NR . 


46.2 

52.3 
1.5 


Comment:  [Note  specific  details  of  this  MSG.] 
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CHARACTERIZATION:  PLANNING  FACTORS  AND  DATA 


#73.  Data  Sources  and  Validity:  Where  did  the  data  for  this  MSG  come 
from?  List  sources  in  as  complete  a  fashion  as  possible.  Were  any 
independent  checks  performed  to  insure  the  accuracy,  timeliness, 
consistency,  and  overall  quality  of  the  data?  Describe  them. 

DATAS,  4/20 

*73-0  DATA  SOURCES  AND  VALIDITY:  [Comments] 

NA;  UNKNOWN  . 

MILITARY;  NO  CROSS  CHECK  . 

MILITARY;  CROSS  CHECK  . 

CIVIL . 

GENERATED  OWN . 

FIELD  EXERCISE  . 

COMBINATIONS  . 

NR . 

#74.  Types  of  Data:  It  has  been  suggested  that  three  types  of  data  can 
be  distinguished  in  games  and  simulations : 

Type  1  data  =  High  certainty  data 

Examples :  range  of  a  weapon  under  specified  conditions ,  the 

size  of  a  unit  of  troops,  etc. 

Type  2  data  -  Certain  level  of  uncertainty 

Examples:  outcome  of  a  company  fight,  radar  detection  range 

(these  need  parametric  studies  and  sensitivity 
analysis  for  validation) . 

Type  3  data  =  High  uncertain  and  hard  to  test 

Examples:  diplomatic  behavior,  enemy  goals,  broad  social 

or  economic  reactions  to  policy. 


3.8 

30.3 

29.5 

1.5 

2.3 

1.5 

25.0 

6.1 


*74-0  TYPES  OF  DATA 

[In  terms  of  the  three  types  of  data  requirements  describe  the  data 
needed  for  your  game  or  simulation.] 

#75.  Number  of  Inputs  (Constants,  Parameters,  and  Variables)  in  MSG. 

This  may  vary  from  use  to  use;  therefore,  if  necessary,  give  over  oun  , 
average,  and  upper  bound. 

75-0  NUMBER  OF  INPUT  CONSTANTS:  (Specify) 

75-1  NUMBER  OF  INPUT  PARAMETERS:  (Specify) 

75-2  NUMBER  OF  INPUT  VARIABLES:  (Specify) 


Low  Middle  High 


> - 1 — 

Confidence  Level 


Comment : 
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#76.  Number  of  Output  Variables  in  MSG:  This  may  vary  from  use  to 
use*  therefore,  if  necessary,  give  lower  bound,  average,  and  upper  bound. 

76-0  NUMBER  OF  OUTPUT  VARIABLES:  (Specify)  _ 


Low  Middle  High 

i _ _ _ \ - -J - — i 

Confidence  Level 


Comment: 

#77.  Intangibles:  Are  sometimes  ruled  out  by  limiting  the  scope  of 
the  study;  by  obtaining  rulings  from  higher  authority  as  to  how  they 
are  to  be  treated;  by  using  expert  estimates;  by  using  high  and  low 
bounding  procedures  or  by  other  methods. 

*77-0  INTANGIBLES 

[Describe  how  intangibles  are  treated  in  this  MSG.  In  answering 
give  a  for  instance.] 

#78.  Sensitivity  Analysis:  Discuss  the  importance  of  sensitivity 
analysis  for  this  MSG  and  describe  how  it  is  done,  if  it  is  done  at  all. 
We  are  not  interested  in  sensitivity  analysis  done  as  a  routine  matter 
of  debugging;  rather,  what  has  been  done  since  the  MSG  has  been 
operational? 


SEN  SET,  4/21 

78-0  SENSITIVITY  ANALYSIS  PERFORMED?:  NO  . .  44.7 

YES  .  46.2 

NR .  9.1 


Comment : 


*79-0  DATA  COLLECTION  TIME:  ( _ _ _ ) 

[Estimate  (in  man-years)  data  collection  time  required.] 

Low  Middle  High 

t _ _ i _ _ 1 - — - 1 

Confidence  Level 

#80.  Data  Validation:  Frequently  all  data  come  from  another  agency 
or  source  with  no  checks  from  the  user  groups .  Sometimes  a  user  group 
obtains  its  information  first  hand  by  measurement,  observation,  field 
tests,  etc.  Describe  how  you  get  your  data  inputs  and  what  independent 
checks  or  procedures  you  perform  to  challenge  the  validity  of  the  data. 
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DATAV,  4/22 

*80-0  DATA  VALIDATION: 


NA;  UNKNOWN .  6.8 

HIGH  QUALITY .  13.6 

MODERATE .  2,8.0 

WEAK .  29.5 

NOT  DONE . 15.9 

NR .  6.1 

\ 


MODEL  OPERATION:  SUPPORTING  FACILITIES 


#81.  Computer  Used  for  Running  a  Simulation:  We  mean  the  different 
computers  for  which  this  model  has  been  run.  In  some  cases  there  may  be 
only  one,  and  in  other  cases  many  modifications  may  have  been  issued 
for  different  machines.  List  not  more  than  the  three  most  frequently 
used  operating  systems. 

81-0  COMPUTER  USED  FOR  RUNNING:  NA 

81-1  COMPUTER(S)  USED  FOR  RUNNING:  THREE  OR  LESS  (Specify) 

Low  Middle  High 

I - 1 - 1 - » 

Confidence  Level 


#82.  Program  Language:  This  calls  for  the  language  in  which  the 
simulation  has  been  programmed.  Frequently  there  may  be  a  series  of 
languages.  All  should  be  noted. 

LANG,  4/23 

82-0  PROGRAM  LANGUAGE:  NA 

82-1  PROGRAM  LANGUAGE (S) :  (Specify) 


NA . 1.5 

FORTRAN .  79.5 

PL-1  .  1-5 

COBOL .  0.8 

GPSS .  3.0 

SIMSCRIPT .  3.8 

ASSEMBLER .  3.0 

Other .  2.3 

NR .  4.5 

NR  Low  Middle  High  Certain 

82CL ,  4/24  i  5.5  ,  0.8  i  6.8  i  44. 7  I  42.4  i 

Confidence  Level 


#83.  Program  Size:  Approximately  how  many  instructions  are  there  in  the 
language (s)  noted  above? 
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MSGSIZ 

83-0  PROGRAM  SIZE:  (Specify  language) 

83-1  PROGRAM  SIZE:  (Specify  language) 

NAj  unknown  . 

<  1000  instructions  . 

1- 2K . 

2- 4K . 

4-6K . 

6-8K . 

8-10K  . 

>  10K . 

NR . 

#84.  Facilities:  #84-4  This  refers  to  the  situation  where  a  special 
system  set  of  languages  or  program  may  have  been  written  to  accompany 
the  running  and  general  handling  of  a  specific  model.  For  example,  some 
models  depend  upon  the  availability  of  much  of  the  specialized  extra 
hardware  and  software.  Although  it  is  possible  that  the  models  them¬ 
selves  can  travel,  much  of  their  power  is  lost  when  the  accompanying 
personnel  and  equipment  are  not  available. 


6.1 

8.3 

9.8 

15.9 

9.8 

7.6 

3.0 

27.3 

12.1 


FACIL ,  4/26 

84-0  FACILITIES:  NA;  UNKNOWN  .  65. 2 

84-1  FACILITIES:  SPECIAL  BUILDING  OR  INSTITUTION  .  2.3 

84-2  FACILITIES:  DEDICATED  COMPUTER  (UNCLASSIFIED)  .  6.8 

84-3  FACILITIES:  DEDICATED  COMPUTER/CLASSIFIED  TAPES  .  9.8 

84-4  FACILITIES:  SPECIAL  LANGUAGE,  LIBRARY  OR  COMPUTATIONAL 

SYSTEM .  22.5 

NR .  Z'° 


Low  Middle  High 


_ I - 1 - — - 1 

Confidence  Level 


SIMULATION/STUDY/MODEL  PRODUCTION  COSTS 


General  caveat  on  building  costs.  The  possibilities  for  obtaining 
close  cost  estimates  for  many  MSGs  are  difficult  in  the  extreme.  This 
is  not  merely  a  problem  of  gathering  information;  it  is  a  problem  of 
correct  conceptualization  of  the  costs  that  should  be  imputed  to  certain 
forms  of  work.  In  particular,  joint  costs  play  an  enormously  important 
role;  thus  it  is  not  an  easy  matter  to  impute  costs  for  items  such  as 
computer  time,  joint  use  of  educational  facilities,  and  so  forth.  In 
this  coding  scheme  we  wish  to  stress  that  the  costing  figures  presented 
should  be  used  with  extreme  caution.  _ _ 
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#85.  Development  Time  is  a  concept  about  which  it  is  difficult  to 
be  both  precise  and  accurate.  For  our  purposes,  we  must  emphasize  that 
the  categories  indicated  are  crude  in  the  extreme.  We  are  trying  to 
indicate  the  elapsed  time  between  the  decision  to  build  a  particular 
model  and  the  first  production  run  of  that  model.  In  many  cases  after 
a  model  has  been  used  once,  development  goes  on  for  many  years.  Thus 
our  criterion  may  be  regarded  as  presenting  a  gross  underestimation  of 
development  time.  Furthermore,  additional  complications  appear  on  occasion 
as  a  model  develops  and  it  may  change  its  name.  A  further  clarification 
of  this  idea,  according  to  our  meaning,  is  the  time  from  the  inception  of 
the  work  on  construction  until  the  first  production  run.  This  is  con¬ 
trasted  with  a  debugging  run;  they  are  not  the  same.  We  specifically 
do  not  consider  further  modification  after  the  first  production  run  has 
taken  place,  even  though  ex  poet  facto ,  the  first  production  run  is  then 
regarded  as  "experimental. " 

DEVTM2 y  4/28 


85-0  DEVELOPMENT  TIME:  ( 

) 

<  3  months  . 

3- 6  months  . . 

7-12  months  . 

13-18  months  . 

19-24  months  . 

25-30  months  . 

31-36  months  . 

37-42  months  . 

>  42  months  . 

NB . 

_  3.8 

_  22.0 

_  25.8 

_  9.8 

_  8.3 

_  3.8 

_  3.0 

_  3.0 

_  5.3 

....  15.2 

NR 

Low  Middle 

High 

Certain 

85CL ,  4/29  | 

i  12.1 

i  8.3  ,  29.5 

1  40.2 

1  9.8  ] 

Confidence 

Level 

#86.  We  contrast  total  man-years  with  professional  man-years.  Under 
category  #86  we  include  graduate  students,  secretarial  help,  program¬ 
ming  assistance  and  any  other  forms  of  voluntary  contribution  of 
time.  These  are  direct  man-years  and  do  not  include  allocation  of 


institutional  administrative  overhead. 

DEVMYlj  4/30 

*86-0  DEVELOPMENT  TIME:  TOTAL  MAN-YEARS:  ( _ ) 

NA;  unknown  . 12.9 

0-1  man-years . . .  21.2 

2-5  man-years  . . 31.8 

6-10  man-years . 8.8 

11-20  man-years  . 8.3 

>  20  man-years  .  6.1 

NB . 9.8 

NR  Low  Middle  High  Certain 

86CL,  4/31  i  18.9  \  11.4  ,  41.7  \  22.0  i  6.1  1 


Confidence  Level 
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#87.  Professional  man-years  used  in  the  development  of  a  model.  Under 
this  category  we  include  both  professional  designers  and  consultants. 

In  many  cases  there  are  also  graduate  students,  additional  helpers, 
ordinary  programming  assistance,  as  well  as  an  enormous  amount  of  office 


staff. 

DVPMY1 3  4/32 

87-0  DEVELOPMENT  TIME:  PROFESSIONAL  MAN-YEARS:  (  ) 

NAj  unknown . . . . . . .  12. 1 

0-1  man-years  .  25.8 

2-5  man-years  . . .  32.6 

6-10  man-years . 9.8 

11-20  man-years . ........ . .  6.8 

>  20  man-years  . . .  2.3 

NR . 10.6 


#88.  Development  Team  Professional  Profile:  Describe  the  professional 
makeup  of  the  development  team  (including  consultants) . 

*88-0  DEVELOPMENT  TEAM  PROFESSIONAL  PROFILE:  (Describe) 

NR  Low  Middle  High  Certain 

88CLJ  4/33  ,  23.5  (  6.8  \  20.5  ^  37.1  \  12.1  i 

Confidence  Level 


DVPRGlj  4/34 

89-0  DEVELOPMENT  TIME:  PROGRAMMER  MAN-YEARS  ( _  ) 

NA;  unknown .  19.7 

0-1  man-years  .  37.1 

2-5  man-years  .  24.2 

6-10  man-years . . .  4.  5 

11-20  man-years  . . 3.0 

>  20  man-years  . 1*5 

NR .  9.8 


MODEL  CHARACTERIZATION :  DOCUMENTATION 


#90.  Documentation:  #90-1  Excellent  means  that  the  documentation  is 
sufficiently  good  that  it  can  be  picked  up  elsewhere  by  a  different 
group  of  people  and  operated  without  or  with  a  minimum  of  long-distance 
telephone  calls  and  conferences.  #90-3  Average  means  that  the 
documentation  exists  in  some  form  but  it  is  moderately  hard  to  operate 
without  at  least  some  discussions  with  the  originators  of  the  document. 
#90-6  Uneven/highly  variable  is  put  in  to  characterize  simulation  in 
which  there  is  spotty  documentation,  often  indicating  an  evolution  of 
different  programmers  and  different  groups  working  on  the  model.  To 
get  decent  documentation  one  may  have  to  search  among  the  disarray  of 
documents  that  are  presented. 
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DOCEXT ,  4/25 


90-0  DOCUMENTATION,  EXTENT:  NA;  ZERO;  UNKNOWN  .  2.2 

90-1  DOCUMENTATION,  EXTENT:  EXCELLENT  .  15.9 

90-2  DOCUMENTATION,  EXTENT:  VERY  GOOD  .  26.5 

90-3  DOCUMENTATION,  EXTENT:  AVERAGE .  22.6 

90-4  DOCUMENTATION,  EXTENT:  WEAK  .  8.2 

90-5  DOCUMENTATION,  EXTENT:  POOR .  1.5 

90-6  DOCUMENTATION,  EXTENT:  UNEVEN/HIGHLY  VARIABLE  .  2.8 

90-7  DOCUMENTATION,  EXTENT:  UNAVAILABLE  .  6.1 

Other  Combinations  .  0.8 

NB .  2.2 


#91.  Documentation  Availability/Location:  91-5.  Proprietary  (classified)/ 
write  author.  What  we  mean  here  is  that  the  information  on  the  simulation 
is  classified  in  the  sense  of  top  secret,  secret,  and  so  on.  To  obtain 
this  information,  it  is  necessary  to  write  the  author.  This  relieves  the 
burden  of  identifying  the  document  and  approving  of  its  transmittal  to 
the  author  and  to  the  people  who  are  searching  for  the  document.  Problems 
of  clearance,  need  to  know,  and  so  on  can  then  be  resolved  between  the  two 
interested  parties. 


DOCL0C1 ,  4/26-28 


91-0 

DOCUMENTATION, 

AVAIL. /LOC. 

91-1 

DOCUMENTATION, 

AVAIL. /LOC. 

91-2 

DOCUMENTATION , 

AVAIL. /LOC. 

91-3 

DOCUMENTATION, 

AVAIL. /LOC. 

91-4 

DOCUMENTATION, 

AVAIL. /LOC. 

91-5 

DOCUMENTATION , 

AVAIL. /LOC. 

91-6 

DOCUMENTATION, 

AVAIL. /LOC. : 

91-7 

DOCUMENTATION, 

AVAIL. /LOC.  : 

91-1 ! 

DOCUMENTATION, 

AVAIL. /LOC. : 

NB 


NA;  UNKNOWN  .  9.8 

OUT  OF  PRINT/UNKNOWN .  6.1 

PROPRIETARY /NOT  FOR  PROFITS  4.5 

PROPRIETARY/COMMERCIAL  .  0.0 

PROPRIETARY /WRITE  AUTHOR  ...  10.6 
PROPRIETARY  (CLASSIFIED)/ 

WRITE  AUTHOR .  16.7 

PUBLIC/DEFENSE  DOCUMENTATION 

CENTER .  47.0 

PUBLIC /LIBRARY  OF  CONGRESS  .  0.0 

PUBLIC /PROFES  S IONAL 

JOURNALS,  BOOKS  .  0.8 

. .  4.5 


*92-0  DOCUMENTATION:  GENERAL  OVERVIEW  (Describe) 


#93.  Publication  Type:  #93-3.  Reports /analyses ,  etc.  This  refers  to 
publications,  possibly  generated  after  a  series  of  runs,  to  be  used  as 
an  official  document,  as  a  report  to  a  higher  authority,  or  possibly  as 
even  a  supporting  argument  for  a  request  for  funds.  This  is  in  distinc¬ 
tion  to  a  document  which  is  a  book  or  article  for  nonspecific  purposes. 


DOCPUB ,  4/29 

93-0  PUBLICATION  TYPE:  BOOKS  OR  ARTICLES  - 

93-1  PUBLICATION  TYPE:  USER  MANUALS  . . 

93-2  PUBLICATION  TYPE:  PROGRAM  DECKS /LISTINGS 
93-3  PUBLICATION  TYPE:  REPORTS /ANALYSES ,  ETC. 

92-0  and  92-1 . . . 

92-1  and  92-2  . 

Other  combinations  . 

NB . 


1.5 

12.6 

2.8 

24.2 

1.5 

20.5 

25.8 

9.1 
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*94-0  PUBLICATION  OR  DOCUMENT  IDENTIFICATION: 

[Specify  one  or  two  documents  most  relevant  to  this  game  simulation 
or  study.  Give  full  references  so  that  documentation  may  be  assembled. ] 


TECHNICAL  COORDINATION  AND  STANDARDS 


#95.  Technical  Coordination:  One  might  have  a  central  clearing  house 
which  performs  a  clerical  operation  with  no  professional  or  evaluative 
role.  One  might  otherwise  have  a  staff  of  several  permanent  professionals 
whose  task  is  to  compose  and  to  technically  describe  the  inventory  of 
models,  simulation,  or  games.  In  your  opinion,  supposing  that  a  central 
clearing  house  exists,  should  it  have  a  technical  staff?  Discuss. 


TECH t  4/40 

95-0  TECHNICAL  COORDINATION:  HIGHLY  UNDESIRABLE  .  12.9 

95-1  TECHNICAL  COORDINATION:  UNDESIRABLE . 12.6 

95-2  TECHNICAL  COORDINATION:  INDIFFERENT  .  21.2 

95-3  TECHNICAL  COORDINATION:  DESIRABLE  .  22.7 

95-4  TECHNICAL  COORDINATION:  HIGHLY  DESIRABLE  .  22.0 

NR .  7.6 


#96.  Standards  Committee:  Question  58  asked  about  the  desirability 
of  standardization  without  specifying  what.  Is  it  premature  to  try 
to  form  a  professional  standards  committee  for  models,  games,  and 
simulations.  Is  it  needed?  Would  it  probably  do  good  or  harm? 
Please  Comment. 


STNRD2 ,  4/41 

HIGHLY  USEFUL  .  0.0 

USEFUL . 14.4 

SAME .  9.1 

HARMFUL  .  41.7 

HIGHLY  HARMFUL  .  20.5 

NR .  14.4 


#97.  Questionnaire  Evaluation  [Written  comments  are  also  welcome.] 

QUEVAL,  4/42 

97-0  RESPONDENT’S  EVALUATION  OF  QUESTIONNAIRE :  EXCELLENT  -  2.2 


•97-1  RESPONDENT'S  EVALUATION  OF  QUESTIONNAIRE:  GOOD  .  20.5 

1 97—2  RESPONDENT'S  EVALUATION  OF  QUESTIONNAIRE:  MODAL .  25.8 

97-3  RESPONDENT'S  EVALUATION  OF  QUESTIONNAIRE:  POOR .  24.2 

97-4  RESPONDENT’S  EVALUATION  OF  QUESTIONNAIRE:  BAD  .  12.9 

NR .  14  -4 

Researchers  '  evaluation  of  the  quality  of  the  questionnaire  responses : 


IQC,  4/42 

EXCELLENT .  2,.  2 

GOOD .  19.7 

MODAL  .  2,2.6 

POOR .  20.2 

BAD . .  I5- 2 
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Part  XXI:  Man-Machine  Gaines  or  Simulations  and  Manual  Games 
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CHARACTERIZATION:  GENERAL  DESCRIPTION 


#98.  Control  Team:  By  the  phrase  "control  team"  we  mean  a  formal  team 
as  part  of  the  game  making  up  rules  or  interacting  with  the  other  teams 
as  the  game  progresses  (#98-3).  This  should  be  contrasted  with  game 
management  control  (#98-2) ,  where  the  directors  or  the  managers  or  referees 
do  not  play  a  direct,  important,  game -influencing  role.  For  example, 
most  business  games  under  this  categorization  do  not  have  a  control 
team.  Few  two-sided  dueling  games  have  control  teams.  Almost  all 
political-military  exercises  have  control  teams. 


98-0 

CONTROL 

TEAM: UNKNOWN 

98-1 

CONTROL 

TEAM:N0 

98-2 

CONTROL 

TEAM:YES ,  BUT  COULD  BE  COMPUTERIZED 

98-3 

CONTROL 

TEAM: YES ,  MUST  BE  USED 

Low  Middle  High 

I _ -J - — J - 1 

Confidence  Level 


#99.  Number  of  Live  Player  Teams;  This  excludes  a  control  team.  If 
the  game  has  been  designed  to  have  a  variable  number  of  teams,  this 
should  be  noted.  The  number  of  live  teams  actually  used  in  different 
runs  should  also  be  indicated. 

99-0  NUMBER  OF  LIVE  PLAYER  TEAMS:  ( _ _  )  Specify 


#100.  Number  of  Robots:  The  same  observations  hold  for  robots  or 
completely  computerized  teams.  In  simulations  where  a  combat  system  is 
being  simulated,  such  as  in  a  totally  computerized  duel,  we  may  regard 
the  model  as  having  two  robots,  one  for  each  side  playing  the  other. 

A  simple  test  for  this  classification  would  be  to  ask  how  to  convert 
this  game  into  a  man-machine  game.  In  order  to  do  so,  some  of  the 
automated  decisionmaking  of  one  or  both  sides  would  be  removed  and 
replaced  by  live  player  decisions. 

100-0  NUMBER  OF  ROBOT  TEAMS:  (  )  Specify 
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#101.  Sequencing  of  Moves:  There  are  some  games  (such  as  many  of  the 
war  games)  played  where  moves  are  simultaneous.  Furthermore,  many  games ^ 
such  as  two-person  matrix  experiments,  usually  utilize  simultaneous  moves. 
There  are  other  games  in  which  the  moves  are  in  fixed  sequential  order; 
examples  of  such  are  chess  or  checkers.  There  are  other  games  in  which 
the  moves  are  in  variable  order;  frequently  either  chance  will  determine 
the  next  move  or  a  player  is  in  a  position  to  give  the  move  to  another 
player.  Craps  is  an  example  of  just  such  a  game;  depending  on  how  one 
defines  chess,  one  pauses  to  see  who  selects  sides  at  the  beginning  in  the 
first  move  in  an  invariable  order,  after  which  it  is  in  fixed  sequential 
order.  Another  set  of  examples  is  sporting  events.  In  baseball  the 
batting  order  is  fixed.  In  football,  the  interteam  play  goes  in  no 
particular  order  although  a  series  of  downs  is  in  fixed  format. 

In  some  games  some  of  the  moves  may  be  simultaneous,  whereas  others  may 
be  in  variable  order.  For  example,  in  some  strategic  war  games  it  may 
be  required  to  pay  costs  for  force  maintenance  every  period.  However,  when 
new  weapons  sy terns  investment  considerations  are  included,  it  is  up  to 
the  individual  team  to  decide  whether  or  not  they  intend  to  invest. 


101-0  SEQUENCING  OF  MOVES: 
101-1  SEQUENCING  OF  MOVES: 
101-2  SEQUENCING  OF  MOVES: 
101-3  SEQUENCING  OF  MOVES: 


UNKNOWN 
SIMULTANEOUS 
FIXED  SEQUENTIAL 
VARIABLE  ORDER 

Low  Middle  High 

t _ 1 _ 1 _ _ _J 

Confidence  Level 


#102.  Moves  per  Team:  In  this  case,  for  complex  games,  we  are  referring 
to  a  move  by  the  team  as  a  whole,  not  to  the  many  individual  small  transac 
tions  that  might  be  taking  place. 

There  are  some  games,  such  as  damage-exchange  rate  and  attrition  evalua¬ 
tion  games  or  continuous  search  games,  in  which  the  simulation  or  the 
computation  is  basically  a  mathematical  procedure  with  no  clear 
definition  of  move.  In  this  instance  we  classify  the  move  description 
102-0. 


102-0  MOVES  PER  TEAM:  NA; UNKNOWN 

102-1  MOVES  PER  TEAM:  MOVES  PER  TEAM  ( _ _ _)  Specify 
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#103.  Complementary  Procedures:  #103-2  includes  lectures;  #103-4 
includes  mathematical  solutions  such  as  game— theoretic  solutions;  and 
#103-6  includes  field  exercises. 


103-0 

COMPLEMENTARY 

PROCEDURES 

103-1 

COMPLEMENTARY 

PROCEDURES 

10  3-2 

COMPLEMENTARY 

PROCEDURES 

103-3 

COMPLEMENTARY 

PROCEDURES 

103-4 

COMPLEMENTARY 

PROCEDURES 

103-5 

COMPLEMENTARY 

PROCEDURES 

103-6 

COMPLEMENTARY 

PROCEDURES 

NA; UNKNOWN 

NONE 

LECTURES 

CASE  STUDIES /HISTORY 

SIMULATION/ANALYSIS 

"EXPERIENCE" 

FIELD  EXERCISE 

Low  Middle  High 

1 _ l - 1 - 1 

Confidence  Level 


104-0 

FORMAL 

GAME 

TYPE 

104-1 

FORMAL 

GAME 

TYPE 

104-2 

FORMAL 

GAME 

TYPE 

104-3 

FORMAL 

GAME 

TYPE 

104-4 

FORMAL 

GAME 

TYPE 

NA;UN KNOWN 

UNDEFINED,  PAYOFFS  NOT  SPECIFIED 
CONSTANT  SUM  GAME 
NON-CONSTANT  SUM  GAME 
ONE-PERSON  MODEL  (MAXIMIZATION) 


Low  Middle  High 


Confidence  Level 
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#105.  Rules:  #105-1  Rigid  manual.  An  example  of  a  rigid  manual  game 
would  be  chess.  All  of  the  rules  are  well  specified  in  advance  and 
the  game  is  a  manual  game.  #105-2  Semi-rigid  manual.  An  example  would 
be  a  war  game,  where,  although  the  fire  power  and  other  planning  fac¬ 
tors  are  supplied,  some  questions  during  the  game  may  be  addressed 
to  a  referring  board  and  certain  rules  or  rulings  are  made  during 
the  course  of  play.  #105-3  Free-form  or  referee’s  direction  would  be 
a  game  such  as  a  political  military  exercise  in  which  the  control  teams 
and  the  referee’s  direction  are  critical  to  conducting  the  game.  The 
melding  of  the  moves  and  adjudication  of  attempted  moves  is  a  critical 
feature  of  such  a  game.  #105-4  Rigid  computerized  rules  are  dis¬ 
tinguished  from  rigid  manual  inasmuch  as  in  general  the  rules  of  manual 
games  are  much  more  visible  and  hence  much  more  open  to  questioning 
than  are  those  of  computerized  games.  One  of  the  major  dangers  of 
using  computerized  games  is  that  a  great  amount  of  bad  modeling  and 
theorizing  can  be  hidden  in  computer  programs.  #105-5  Semi-rigid 
rules  computerized.  In  this  case  the  game  may  be  computerized,  but 
not  all  of  the  rules  are  necessarily  described.  For  example,  one 
might  have  a  business  or  marketing  game  in  which  although  virtually 
everything  to  do  with  sales,  manufacturing,  and  internal  running  of 
the  firm  has  been  computerized,  press  releases  and  newspapers  are 
issued  to  the  firm,  thus  adding  a  verbal  and  somewhat  less  formalized 
component  to  the  game.  #105-6  In  some  instances  games  or  simulations 
are  nothing  more  than  the  dynamics  of  the  behavior  of  a  formal  mathe¬ 
matical  model  or  computer  program.  The  category  also  includes  rules 
that  are  well  defined  in  a  game- theoretic  sense. 

In  some  games,  although  all  rules  are  given  they  are  so  complicated  that 
no  single  individual  will  know  all  of  them  (#105-7) .  In  other  games 
part  of  the  purpose  is  to  discover  unstated  rules  (#105-8) .  In  free¬ 
form  games,  there  may  be  considerable  uncertainty  concerning  basic 
structure  (#105-9) . 


105-0 

RULES : 

NA;UNKNOWN 

105-1 

RULES: 

RIGID  MANUAL 

105-2 

RULES: 

SEMI-RIGID  MANUAL 

105-3 

RULES: 

FREE  FORM/ REFEREE'S  DIRECTION 

105-4 

RULES: 

RIGID  COMPUTERIZED 

105-5 

RULES: 

SEMI-RIGID  COMPUTERIZED 

105-6 

RULES: 

PROGRAM  OR  FORMAL  MATH  MODEL 

105-7 

RULES: 

WELL  DEFINED  BUT  TOO  LARGE  FOR  COMPREHENSION 

105-8 

RULES: 

SOME  RULES,  NOT  KNOWN 

105-9 

RULES: 

UNCERTAINTY  CONCERNING  BASIC  STRUCTURE 

Low  Middle  High 

1 _ l _ J - -J 

Confidence  Level 
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#106.  Structure  of  Game:  In  the  category  structure  of  games  we  include, 
under  #106-3  matrix  game,  a  game  which  could  conceivably  be  approximated 
by  a  matrix  even  though  in  some  cases  the  strategies  may  be  continuous. 

For  example,  it  may  be  permitted  to  have  a  player  pick  any  price  he 
wants  in  the  range  from  $0  -  $100,  but  in  general  he  may  be  limited  to 
bids  in  units  of  a  dollar.  Even  if  he  wanted  to  make  it  a  continuous 
game,  it  is  quite  possible  that  the  machine  would  round  it  off  by  a 
finite  approximation  and  thereby  make  it  a  matrix  game. 

Some  games  may  fit  into  more  than  one  category.  For  example,  a  business 
game  may  have  a  formal  structure  such  as  an  iterated  matrix  game;  however, 
it  begins  play  with  a  scenario  describing  the  state  of  the  market. 

#106-2  Explicit  mathematical  2x2  matrix  games.  #106-3  Other  matrix 
game.  Here  we  are  referring  to  a  purely  abstract  mathematical  structure 
provided  as  the  venue  for  the  game.  #106-4  Implicit  computerized  damage- 
exchange  calculation  which  in  fact  can  be  regarded  as  a  computation  on 
an  enormously  large  matrix.  #106-5  Iterated  matrix  game.  In  many 
instances,  such  as  many  plays  with  the  Prisoner's  Dilemma,  the  game  is 
played  in  a  dynamic  mode.  However,  a  great  amount  of  the  decisionmaking 
is  performed  on  the  same  structure  period  after  period.  Many  business 
games  have  the  same  "battlefield”  of  a  more  or  less  similar  market  each 
period.  #106-6  Free  form  extensive  .  Games  like  political  military 
exercises  would  fall  in  this  category.  They  are  played  move  by  move, 
but  they  are  not  necessarily  repeating  the  same  situation  on  each  move. 

#106-7  Formal  extensive.  A  game  such  as  chess  is  played  in  formal 
extensive  manner.  The  rules  are  rigid  and  well-defined;  however,  the 
players  move  play  by  play  and  do  not  enunciate  overall  strategies  for 
this  game.  The  remaining  two  categories  concern  games  in  characteristic 
function  form  or  games  in  which  the  prime  area  of  investigation  is  coalitions. 
#106-8  Free  form  characteristic  function.  Such  a  game  would  be  one 
devoted  to  studying  the  coalition  possibilities  for  a  treaty  on  the  use 
of  the  Danube,  as  just  one  example.  #106-9  Formal  characteristics 
function  form.  There  have  been  experiments  done  using  games  specified 
in  characteristic  function  form  where  the  experimenter  studies  how  the 
players  divide  the  money  they  receive  from  cooperative  acts. 

Question  #106  calls  for  some  familiarity  with  several  concepts  of  game 
theory.  If  you  are  not  sufficiently  familiar  with  the  terms  to  answer, 
check  #106-11  and  proceed  to  the  next  question. 

106-0  STRUCTURE  OF  GAME:  UNKNOWN/NA 

106-1  STRUCTURE  OF  GAME:  j SCENARIO/ VERBAL  DESCRIPTION 

106-2  STRUCTURE  OF  GAME:  2X2  MATRIX 

106-3  STRUCTURE  OF  GAME:  OTHER  MATRIX 

106-4  STRUCTURE  OF  GAME:  IMPLICIT  MATRIX 

106-5  STRUCTURE  OF  GAME:  ITERATED  MATRIX 

106-6  STRUCTURE  OF  GAME:  "FREE  FORM  EXTENSIVE" 

106-7  STRUCTURE  OF  GAME:  FORMAL  EXTENSIVE 

106-8  STRUCTURE  OF  GAME:  "FREE  FORM"  CHARACTERISTIC 

106-9  STRUCTURE  OF  GAME:  FORMAL  CHARACTERISTIC  FUNCTION 

106-10  STRUCTURE  OF  GAME:  PURE  MATH  MODEL:  SIMULATED  OR  ANALYZED 

106-11  {STRUCTURE  OF  GAME:  QUESTION  NOT  CLEAR  Low  Middle  High 


Confidence  Level 
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#107.  Information  State:  #107-1  and  #107-2/The  first  refers  to  perfect 
information  in  the  game- theoretic  sense:  all  players  know  all  that  can 
be  known  at  all  times.  An  example  of  a  game  with  perfect  information 
is  a  chess  game.  Incomplete  information  is  the  situation  that  prevails 
in  a  poker  game.  The  kibitzers  or  a  referee  may  know  what  the  hands  of 
all  the  players  look  like,  but  the  players  do  not  know  each  other* s  hands. 
#107-5  Considerable  structural  uncertainty  refers  to  games  in  which  the 
rules  and  the  general  environment  are  by  no  means  clear  at  the  beginning 
of  the  game.  An  example  would  be  an  extremely  free  form  political  game 
to  be  played  in  a  future  time  period  with  the  skimpiest  of  scenarios 
available.  #107-3,  #107-6  In  some  games  that  are  designed  to  teach 
procedures  and  approaches  to  a  problem,  it  is  worth  distinguishing 
information  states  in  which  information  can  be  bought  as  contrasted  with 
those  in  which  calculation  can  be  bought.  For  example,  in  a  business 
game  one  may  have  information  concerning  the  reaction  of  the  market  to 
various  arrays  of  prices.  On  the  other  hand,  one  may  not  have  procedures 
for  fitting  models  to  this  information.  The  procedures  such  as  least- 
square  statistical  packages  can  be  regarded  as  calculation  packages.  In 
some  cases  these  may  be  available  to  players  from  ’’consultants  who  charge 
for  their  use. 


107-0 

INFORMATION 

STATE: 

107-1 

INFORMATION 

STATE : 

107-2 

INFORMATION 

STATE : 

107-3 

INFORMATION 

STATE : 

107-4 

INFORMATION 

STATE: 

107-5 

INFORMATION 

STATE: 

107-6 

INFORMATION 

STATE : 

#108. 

Computer  Use:  Und 

NA;  UNKNOWN 


INFORMATION  CAN  BE  BOUGHT 
SOME  RULES  NOT  KNOWN /MIXED 
CONSIDERABLE  STRUCTURAL  UNCERTAINTY 
CALCULATION  CAN  BE  BOUGHT 

Low  Middle 


High 
J 


Confidence  Level 


somewhat  less  than  one  might  wish  to  describe  as  heavy  analysis.  #108-6 
Man-machine  on-line  interrogative  mode.  We  distinguish  this  from  man- 
machine  interactive  in  the  sense  that  interactive  merely  implies  that  the 
machine  does  the  computations  on  the  environment,  whereas  interrogative 
implies  that  in  the  ;process  of  calculating^  the  machine  questions  the 
player  and  obtains  answers  from  the  player. 


108-0 

COMPUTER  USE: 

NONE/BOARD/FIELD,  ETC. 

108-1 

COMPUTER  USE: 

BOOKKEEPING/LIGHT  STAFF  WORK 

108-2 

COMPUTER  USE: 

ANALYTICAL  AID  TO  PLAY 

108-3 

COMPUTER  USE: 

ANALYTICAL  AID  DEBRIEFING/POST  GAME  ANALYSIS 

108-4 

COMPUTER  USE: 

ANALOGUE 

108-5 

COMPUTER  USE: 

MAN-MACHINE  INTERACTIVE 

108-6 

COMPUTER  USE: 

MAN -MACHINE  ON-LINE  (INTERROGATIVE  MODE) 

108-7 

COMPUTER  USE: 

OTHER  (Specify) 

Low  Middle  High 


Confidence  Level 
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#109  Gaming  Facilities:  #109-7  This  refers  to  the  situation  where  a 
special  system  set  of  languages  or  program  may  have  been  written  to 
accompany  the  running  and  general  handling  of  a  specific  game.  or 
example,  some  games  run  at  the  labs  at  Berkeley  and  some  games  run  with 
the  TRACE  system  at  SDC  or  at  UCLA  depend  upon  the  availability  ot 
much  of  the  specialized  extra  hardware  and  software.  Although  it 
is  possible  that  the  games  themselves  can  travel,  much  of  their  power 
is  lost  when  the  accompanying  programs  and  equipment  are  not  aval  a  e. 

NA;  UNKNOWN 

SPECIAL  BUILDING  OR  INSTITUTION 
SPECIAL  LAB 
DEDICATED  COMPUTER 
RENTED  LAB 
RENTED  "SPACE" 

TEMPORARY  "FREE  SPACE" ;  INFORMAL 

SPECIAL  LANGUAGE,  LIBRARY  OR  COMPUTATIONAL  SYSTEM 


109-0 

GAMING 

FACILITIES : 

109-1 

GAMING 

FACILITIES : 

109-2 

GAMING 

FACILITIES : 

109-3 

GAMING 

FACILITIES : 

109-4 

GAMING 

FACILITIES : 

109-5 

GAMING 

FACILITIES : 

109-6 

GAMING 

FACILITIES : 

109-7 

GAMING 

FACILITIES : 

Low 
4 - 


Middle 


Confidence  Level 


High 


GAME  OPERATION  TIME  FRAME 


110-0  SET-UP  TIME  ( _ _)  Specify 

#111.  Elapsed  time  of  run,  start  to  finish:  We  refer  to  the  playing  time 
of  a  single  game  or,  in  the  case  of  experimental  games,  of  a  single  expen 
ment  which  could  involve  several  replications;  for  example,  when  a  series 
of  experiments  is  run  sequentially  over  several  days.  In  some  instances 
the  nature  of  the  game  and  its  format  make  this  figure  quite  precise. 

In  other  cases  there  is  a  variability  of  several  orders  of  magnitude 
concerning  how  long  the  game  takes  to  play.  For  example  some  business 
games  are  run  on  the  basis  of  one  decision  a  day  or  one  decision  a  week 
that  is  made  in  less  than  an  hour;  otherwise,  the  individuals  carry 
on  their  normal  routines. 

Under  this  category  we  refer  to  total  elapsed  time  from  the  start  of  the 
game  to  the  debriefing.  In  some  informal  instances,  such  as  running  a 
game  with  a  class,  one  may  run  the  game  for  the  whole  of  a  semester,  for 
example,  the  Carnegie  Tech  game  may  run  for  a  period  of  seven  or 
months.  The  games  of  the  Studies,  Analysis  and  Gaming  Agency  are  y 

run  in  two  different  modes;  One  is  a  3-1/2  day  intensive  game,  and  t 
other  stretches  over  several  weeks.  In  the  case  of  experiments  games, 
games  are  often  run  in  parallel  ~  possibly  intensively  during  one  evening 
for  the  whole  of  a  game,  but  in  some  cases  batches  of  games  are  run  D'mp 
sing  an  experiment.  In  other  cases  players  may  run  for  more  than  one  day. 

If  a  game  is  run  on  more  than  one  mode,  the  different  elapsed  times 
be  indicated. 


lll-o  ELAPSED  TIME  OF  RUN,  START  TO  FINISH: 


Low  Middle  High 

Confidence  Level 
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#112.  Player  Game  Play  Time:  By  this  we  mean  the  amount  of  time  spent 
in  actually  playing  a  game.  This  includes  briefing,  decisionmaking,  and 
debriefing  associated  with  the  game. 


112-0  PLAYER  GAME  PLAY  TIME 
112-1  PLAYER  GAME  PLAY  TIME 
112-2  PLAYER  GAME  PLAY  TIME 
112-3  PLAYER  GAME  PLAY  TIME 
112-4  PLAYER  GAME  PLAY  TIME 
112-5  PLAYER  GAME  PLAY  TIME 
112-6  PLAYER  GAME  PLAY  TIME 


NA; UNKNOWN 
<3  HOURS 
>3-6  HOURS 
>6-12  HOURS 
>12-24  HOURS 
>1-7  DAYS 

>1  WEEK  ( _ )  [Specify] 


Low  Middle  High 


l _ i _ ! _ 

Confidence  Level 


#113.  Formal  Game  Prebriefing  Procedure:  We  note  that  the  military  uses 
the  word  "indoctrination"  when  describing  materials  sent  out  prior  to 
the  formal  briefing  time  in  a  game. 


The  distinction  to  be  made  here  concerns  whether  or  not  a  game  has  a 
formal  prebriefing  procedure  or  whether  the  prebriefing  is  informal  or 
nonexistent.  For  example,  chess  players  in  general  need  no  prebrief  mg 
if  they  already  know  the  rules.  Some  simple  games,  such  as  experiments 
with  2x2  matrix  games,  may  be  run  with  an  informal  briefing  from  the 
experimenter  who  has  either  decided  to  dispense  with  formal  control  or 
has  overlooked  the  use  of  formal  control  in  the  verbal  description  of 
the  game. 


113-0  FORMAL  GAME  PREBRIEFING  PROCEDURE: 
113-1  FORMAL  GAME  PREBRIEFING  PROCEDURE: 
113-2  FORMAL  GAME  PREBRIEFING  PROCEDURE: 


NA; UNKNOWN 

NO 

YES 
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High 
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#114  .  Formal  Briefing  Time  (%)  :  This  can  be  described  as  a  percentage 
of  player  game-play  time.  For  example,  if  it  takes  a  player  ten  hours 
to  play  a  game  and  there  is  a  briefing  session  of  1/2  an  hour,  this 
means  that  briefing  time  is  five  percent  of  game  play  time. 

114-0  FORMAL  BRIEFING  TIME  (%)  :  C  _ _ )  [Specify] 

Low  Middle  High 
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#115.  Debriefing  Time:  This  is  also  stated  as  a  percentage  of  game 
play  time  for  the  player.  In  general,  many  experimental  games,  and 
certainly  games  for  entertainment,  have  little  if  any  debriefing. 
Occasionally  there  are  post  mortems  after  chess.  Operational  games  and 
games  for  teaching  and  training  may  have  considerable  debriefing.  For 
operational  use,  the  length  of  debriefing  is  fairly  clear;  for  example, 
after  a  SAGA  game,  there  may  be  a  half  day  set  aside  (beyond  the  three 
days  of  play)  specifically  for  discussion  and  formal  debriefing.  How¬ 
ever,  with  a  game  used  for  teaching  purposes,  such  as  the  Carnegie  Tech 
game,  one  might  regard  the  complete  course  taken  with  the  game  as  a 
briefing-debriefing  session;  in  that  case,  one  could  claim  that  the 
debriefing  and  briefing  time  could  easily  be  as  large  if  not  larger  than 
the  game-playing  time. 

If  a  game  is  used  for  different  purposes  one  should  indicate  the  briefing 
and  debriefing  time  of  each.  The  context  of  purposes  stated  in  the  earlier 
question  should  make  clear  the  specific  category  to  which  a  game  belongs • 

115-0  DEBRIEFING  TIME  (%)  :  >25  ( _  )  [Specify] 

Low  Middle  High 
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#116.  Control  Time,  Total  Expenditure:  This  includes  briefing  time, 
running  time,  and  debriefing  time  of  the  control  group.  For  example, 
in  a  SAGA  exercise,  there  may  be  field  trips  and  so  forth  before  the 
game’s  scenario  can  be  written.  This  type  of  work  would  be  classified 
under  game  construction.  The  time  we  are  interested  in  here  is  that 
amount  spent  by  the  individuals  composing  the  control  team  for  running 
purposes.  If  it  is  necessary  for  members  of  the  control  team  to  be 
briefed  or  indoctrinated  for  several  weeks  in  advance,  this  would  be 
counted  as  part  of  control  time.  If,  however,  as  is  usually  the  case, 
they  join  the  group  merely  a  day  or  two  or  even  less  before  play  time, 
we  would  start  to  count  control  team  time  from  this  point.  In  some 
instances  there  is  not  a  formal  control  team;  however,  there  is  never¬ 
theless  a  game  director  —  formal  or  informal  —  whose  time  is  being 
used  to  supervise  the  process.  This  should  also  be  counted  even  though 
this  will  generally  amount  to  no  more  than  a  few  hours  or  a  few  days . 

116-0  CONTROL  TIME,  TOTAL  EXPENDITURE:  MAN-WEEKS  ( _ ) 

[Specify] 

Low  Middle  High 
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#117.  Post  Debriefing  Analysis  (Intensity):  This  refers  to  the 
analysis  of  the  game  run  after  the  game  is  over;  the  debriefing  may 
have  taken  place.  In  other  words,  this  should  not  be  confused  with 
debriefing.  It  refers  to  the  analysis  which  may  be  done  by  researchers, 
possibly  the  players  in  a  different  mode,  or  others  to  determine  what 
has  been  learned  from  the  game.  In  the  case  of  experiments,  this  is 
quite  obviously  where  much  of  the  work  is  concentrated.  In  the  case  of 
operational  games,  this  is  where  much  of  the  work  should  be  concentrated 
if  one  wishes  to  measure  the  effectiveness  of  the  exercise.  #117—0 
Post  debriefing  analysis.  For  straight  simulations  there  is  no  debrief¬ 
ing,  hence  this  category  is  not  applicable.  At  the  same  time  simulations 
invariably  involve  analysis  after  they  are  run;  this  is  picked  up  in 
#118.  #117-4,  Considerable,  would  apply  where  the  analysis  time  may  be 

even  more  than  the  game-playing  time. 


117-0 

POST 

DEBRIEFING 

117-1 

POST 

DEBRIEFING 

117-2 

POST 

DEBRIEFING 

117-3 

POST 

DEBRIEFING 

117-4 

POST 

DEBRIEFING 

ANALYSIS  (INTENSITY): 
ANALYSIS  (INTENSITY): 
ANALYSIS  (INTENSITY): 
ANALYSIS  (INTENSITY): 
ANALYSIS  (INTENSITY) : 


NA; UNKNOWN 

NONE 

SLIGHT 

MODERATE 

CONSIDERABLE 

Low  Middle  *  High 
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#118.  Analysis  Time:  Answer  this  question  in  terms  of  man-weeks,  or 
the  percentage  of  actual  time  spent  by  individuals  involved  in  the 
analysis  as  compared  with  total  game  play  time.  There  may  be  an  enormous 
amount  of  automated  analysis  going  on  with  a  small  expenditure  of  human 
time.  This  question  is  concerned  with  the  human  time. 

118-0  ANALYSIS  TIME:  ( _ )  [Specify] 


119-0  SET  UP  COSTS  AS  %  OF  TOTAL  COSTS  TO  OPERATE:  ( _ ) 

[Specify] 

Low  Middle  High 
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#120.  Cost  to  Operate  at  New  Location:  Except  for  going  to  the  new 
location  we  assume  that  costs  will  be  distributed  the  same  way.  In 
other  words,  if  one  is  calculating  on  free  secretarial  help  at  one 
place,  one  calculates  some  free  secretarial  help  at  the  other  place. 

It  makes  a  great  difference  whether  or  not  you  can  bring  in  an  operating 
crew.  However  this  means  that  cost  to  operate  should  be  looked  at  as 
the  minimum  cost  to  bring  in  an  operating  crew  or  cost  to  train  new 
people  on  location*  If  no  crew  is  available,  it  may  be  either  impos¬ 
sible  to  transfer  the  game  or  inordinately  expensive. 

When  we  refer  to  new  location  we  assume  that  the  new  location  has  hard¬ 
ware  that  is  suited  for  the  game  involved. 

120-0  COST  TO  OPERATE  AT  NEW  LOCATION:  ( _ _)  [Specify] 

Low  Middle  High 
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GAME  CHARACTERIZATION:  PLAYERS 


#121.  Player  Selection.  #121-5  "Proprietary  interest  advocates"  refers 
to  the  use  of  players  who  have  a  personal  interest  in  the  actual  use 
of  the  game  or  in  the  use  of  the  results  of  the  game  for  some  specific 
purpose.  For  example,  a  proprietary  interest  advocate  might  be  a  group 
of  individuals  either  advocating  or  opposing  a  weapons  system,  such 
as  MIRV,  or  the  SST,  or  a  specific  piece  of  hardware.  The  game  may 
be  used  as  part  of  an  ongoing  advocacy  process.  In  cases  such  as 
this,  it  is  extremely  important  to  sort  out  players  whose  play  sur¬ 
rounds  the  environment  of  the  game  from  players  whose  interests  cease 
with  the  use  of  the  game  for  whatever  its  explicit,  stated  purposes. 


121-0 

PLAYER 

SELECTION 

121-1 

PLAYER 

SELECTION 

121-2 

PLAYER 

SELECTION 

121-3 

PLAYER 

SELECTION 

121-4 

PLAYER 

SELECTION 

121-5 

PLAYER 

SELECTION 

UNKNOWN 

UNPAID  INDIVIDUAL  VOLUNTEER 
VOLUNTARY  GROUP 
PAID  VOLUNTEER 
COURSE  REQUIREMENT 
"PROPRIETARY  INTEREST  ADVOCATES" 


Low  Middle 


High 
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#122.  Player  Characterization:  Postgraduate  refers  to  individuals  at 
a  war  college  or  other  academic  institution.  #122-3  "Professional"  ap¬ 
plies  to  the  context  of  the  game  being  played;  e.g.,  a  military  man 
playing  a  war  game  would  be  regarded  as  a  professional.  If  he  were 
playing  a  business  game,  he  would  not,  in  general,  be  regarded  as  a 
professional . 


122-0 

PLAYER 

CHARACTERIZATION : 

UNKNOWN 

122-1 

PLAYER 

CHARACTERIZATION: 

POSTGRADUATE 

122-2 

PLAYER 

CHARACTERIZATION: 

ADULT  (NONPROFESSIONAL) 

122-3 

PLAYER 

CHARACTERIZATION : 

PROFESSIONAL 

122-4 

PLAYER 

CHARACTERIZATION: 

OTHER 

Low  Middle  High 
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//123.  Player  Use.  In  some  man-machine  exercises  live  players  are  used 
only  because  they  are  cheaper  or  more  readily  available  than  a  simu¬ 
lated  player.  There  is  no  attempt  to  train  them  nor  are  their  goals  of 
particular  concern  to  the  exercise.  In  this  sense  they  are  merely  a 
substitute  for  machinery;  this  possibility  is  described  in  #123-0. 

123-0  PLAYER  USE:  ONLY  AS  "MACHINERY" 

123-1  PLAYER  USE:  AS  PLAYERS 

[If  answer  to  #123  is  #123-0,  skip  the  remaining  questions  on  players 
and  go  to  #135 . ] 


124-0 

PLAYER 

PAYOFFS : 

124-1 

PLAYER 

PAYOFFS : 

124-2 

PLAYER 

PAYOFFS : 

124-3 

PLAYER 

PAYOFFS : 

124-4 

PLAYER 

PAYOFFS : 

124-5 

PLAYER 

PAYOFFS : 

124-6 

PLAYER 

PAYOFFS : 

UNKNOWN 
MONEY  WAGE 
GRADES  OR  PAYMENT 
FIXED  PRIZE 

PRIZE  PROPORTIONAL  TO  PERFORMANCE 
"EDUCATION" 

NOT  SPECIFIED 


#125.  Player  Pretest  Comprehension  Test:  Is  a  check  made  to  see  if 
the  players  fully  comprehend  the  game  prior  to  play?  This  includes 
the  use  of  a  practice  play  followed  by  questions. 

125-0  PLAYER  PRETEST  COMPREHENSION  TEST:  UNKNOWN 
125-1  PLAYER  PRETEST  COMPREHENSION  TEST:  NO 
125-2  PLAYER  PRETEST  COMPREHENSION  TEST:  YES 

Low  Middle  High 


Confidence  Level 
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#126.  Player  Pretest:  #126-1  refers  to  the  case  where  no  particular 
pretesting  after  the  selection  of  the  players  has  been  performed; 

#126-2  refers  to  games  in  which  one  has  in  fact  run  pretests  on  the 
players.  This  may  be  a  California  Personality  Inventory,  it  may  be 
IQ-tests,  and  so  forth.  One  runs  subjects  or  players  through  a  bar¬ 
rage  of  tests  outside  of  the  formal  game. 

Player  pretest  could  be  a  test  for  IQ,  general  knowledge,  etc.,  or 
some  sort  of  personality (test .  It  does  not  necessarily  imply  a  compre¬ 
hension  test  for  the  game. 

126-0  PLAYER  PRETEST:  UNKNOWN 

126-1  PLAYER  PRETEST:  NO 

126-2  PLAYER  PRETEST:  YES  (Describe) 

Low  Middle  High 

i _ i _ l _ i 

Confidence  Level 


127-0  PLAYER  POST  PLAY  COMPREHENSION  CHECK:  NA; UNKNOWN 

127-1  PLAYER  POST  PLAY  COMPREHENSION  CHECK:  NO 

127-2  PLAYER  POST  PLAY  COMPREHENSION  CHECK:  YES 

Low  Middle  High 


Confidence  Level 
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#128.  Player  Perception  of  Success  of  Purpose:  In  this  characteri¬ 
zation  we  are  not  asking  the  question  whether  the  game  was  effective, 
but  how  the  experience  was  perceived  by  the  players.  A  good  question 
that  must  be  asked  of  all  games  is,  "How  does  enjoyment  correlate 
with  the  value  of  a  game?"  It  is  conjectured  by  us  that  up  to  a 
certain  level  there  is  probably  a  positive  correlation  between  the 
effectiveness  of  a  game  and  the  level  of  enjoyment  or  enthusiasm. 

Beyond  a  certain  level,  however,  we  suspect  that  the  correlation  weakens 
or  goes  negative.  A  highly  enjoyable  game  may  in  fact  have  been 
enjoyed  as  a  game  and  not  as  an  operational,  research,  or  teaching 
device.  Informally  we  have  observed  that  apparently  there  is  not  a 
great  amount  of  correlation  between  highly  popular  lecturing,  acting 
performances,  and  the  amount  of  information  that  is  conveyed  to  the 
students.  In  some  cases  player  reaction  may  be  mixed.  You  may  wish 
to  give  a  percentage  breakdown  next  to  the  categories  or  draw  a  small 
graph. 

128-0  PLAYER  PERCEPTION  OF 
128-1  PLAYER  PERCEPTION  OF 
128-2  PLAYER  PERCEPTION  OF 
128-3  PLAYER  PERCEPTION  OF 
128-4  PLAYER  PERCEPTION  OF 
128-5  PLAYER  PERCEPTION  OF 


Confidence  Level 


SUCCESS  OF  PURPOSE: 
SUCCESS  OF  PURPOSE: 
SUCCESS  OF  PURPOSE: 
SUCCESS  OF  PURPOSE: 
SUCCESS  OF  PURPOSE: 
SUCCESS  OF  PURPOSE: 


NOT  RELEVANT;  UNKNOWN 

HIGHLY  POSITIVE 

POSITIVE 

NEUTRAL 

NEGATIVE 

HIGHLY  NEGATIVE 

Low  Middle  High 


129-0 

"AVERAGE" 

PLAYER'S 

ENJOYMENT : 

NOT  RELEVANT;  UNKNOWN 

129-1 

"AVERAGE" 

PLAYER’S 

ENJOYMENT : 

HIGHLY  POSITIVE 

129-2 

"AVERAGE" 

PLAYER'S 

ENJOYMENT : 

POSITIVE 

129-3 

"AVERAGE" 

PLAYER'S 

ENJOYMENT : 

NEUTRAL 

129-4 

"AVERAGE" 

PLAYER'S 

ENJOYMENT : 

NEGATIVE 

129-5 

"AVERAGE" 

PLAYER'S 

ENJOYMENT : 

HIGHLY  NEGATIVE 
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#130.  Number  of  Players  Per  Team:  Do  not  fill  in  more  than  the  three 
most  frequently  used  categories. 

130-0  NUMBER  OF  PLAYERS  PER  TEAM:  (Specify) 


Low  Middle  High 

1 _ L _ I - ' 

Confidence  Level 


131-0  REPEATED  USE  OF  PLAYERS: 
131-1  REPEATED  USE  OF  PLAYERS: 
131-2  REPEATED  USE  OF  PLAYERS: 


NA;UNKN0WN 

NO 

YES 


Low  Middle  High 

i _ -1 - 1 - - - 1 

Confidence  Level 


#132.  Role  Playing  (Individual):  #132-1  Role  playing:  self  is  the 
category  which  indicates  that  no  other  role  playing  is  required  of  the 
players  in  the  particular  game.  #132-3  Role  playing:  specific  person 
implies  for  example  that  somebody  play  Stalin  or  Mao  Tse  tung  or 
Mr.  Nixon.  #132-4  and  #132-5  A  specific  organization  might  be  something 
like  General  Electric;  an  abstract  organization  would  be  a  large  busin 
firm. 


132-0  ROLE  PLAYING  (INDIVIDUAL): 
132-1  ROLE  PLAYING  (INDIVIDUAL): 
132-2  ROLE  PLAYING  (INDIVIDUAL): 
132-3  ROLE  PLAYING  (INDIVIDUAL) : 
132-4  ROLE  PLAYING  (INDIVIDUAL) : 
132-5  ROLE  PLAYING  (INDIVIDUAL) : 


NOT  RELEVANT /UNKNOWN 
SELF 

SPECIFIC  POSITION 
SPECIFIC  PERSON 

SPECIFIC  ORGANIZATION/INSTITUTION 
ABSTRACT  ORGANIZATION/INSTITUTION 

Low  Middle  High 

| _ l - 1 - - - 

Confidence  Level 


133-0  ROLE  PLAYING  (TEAM)  : 
133-1  ROLE  PLAYING  (TEAM) : 
133-2  ROLE  PLAYING  (TEAM)  : 
133-3  ROLE  PLAYING  (TEAM): 
133-4  ROLE  PLAYING  (TEAM): 
133-5  ROLE  PLAYING  (TEAM)  : 


NOT  RELEVANT /UNKNOWN 
SELF 

SPECIFIC  ORGANIZATION 
ABSTRACT  ORGANIZATION 
SPECIFIC  INSTITUTION 
ABSTRACT  INSTITUTION 

Low 


Middle  High 


Confidence  Level 
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# 134.  Importance  of  Unstated  Purposes:  This  question  involves  the 
characterization  of  players  and  what  might  be  described  as  ”the  game 
outside  of  the  game/1  For  example,  in  the  case  of  a  business  game 
where  many  executives  from  the  same  firm  play  unaccustomed  roles  in  a 
simulated  hierarchy,  there  may  be  pressures  exerted  on  the  individuals 
as  a  result  of  their  being  aware  that  they  are  being  watched.  Even  in 
experimental  games,  the  players  sometimes  may  decide  to  play  ”fool  the 
experimenter”  or  "give  him  what  he  wants.”  This  question  is  admittedly 
subjective,  but  it  merits  serious  consideration. 


134-0  IMPORTANCE  OF  UNSTATED  PURPOSES: 
134-1  IMPORTANCE  OF  UNSTATED  PURPOSES: 
134-2  IMPORTANCE  OF  UNSTATED  PURPOSES: 
134-3  IMPORTANCE  OF  UNSTATED  PURPOSES: 
134-4  IMPORTANCE  OF  UNSTATED  PURPOSES: 

Comments  or  Discussion: 


NA 
HIGH 
SOMEWHAT 
SLIGHT 
NONE 


Low 


Middle 


High 


_j _ J _ 

Confidence  Level 


// 135.  Game  Users:  Count  the  number  of  institutions  where  a  game,  sim¬ 
ulation,  or  a  direct  variant  is  being  used.  This  may  tend  to  produce 
some  overestimation,  yet  for  most  purposes  this  is  the  most  relevant 
figure.  It  gives  an  insight  into  how  widespread  the  direct  use  of  a 
game  has  been  or  how  widespread  the  influence  of  a  game  has  been. 

135-0  GAME  USERS:  (Specify) 


Low  Middle  High 

J - 1— - — * — - - 

Confidence  Level 
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Part  IV:  Voluntary  Assessment 
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Completion  of  this  section: 
VOL  AS,  4/44 

Completed  . 

Did  not  complete  . 


For  the  remaining  questions  assume  that  a  senior  professional 
(one  who  really  knows  the  business)  wishes  to  use  this  MSG  and 
wants  your  evaluation  along  several  dimensions • 

#136.  Assessment  —  Design  and  Construction:  What  are  the  strengths 
and  weaknesses  of  this  MSG’s  design  and  construction? 

Comment: 


#137.  Assessment  -  Planning  Factors  and  Data:  What  in  your  opinion 
are  the  strengths,  weaknesses,  and  constraining  features  of  the  data 
used  in  the  MSG?  How  serious  are  the  deficiencies  or  weaknesses, 
if  any? 


Comment : 
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#138.  Assessment  -  Documentation:  How  complete  and  useful  is  the 
supporting  documentation?  Would  it  be  easy  for  some  other  agency 
to  use  the  MSG,  or  would  the  extent  and  quality  of  documentation 
make  this  difficult  or  impossible? 

Comment: 


#139.  Assessment  -  Operation:  Are  there  peculiarities  of  operation 
that  a  prospective  user  should  be  aware  of?  Is  the  MSG  easy  to  operate, 
or  are  there  unique  procedural  problems  that  one  should  know  about? 

Comment : 
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[For  Man-Machine  or  Manual  Gaines  Only.] 

#140.  Assessment  -  Post  Debriefing  Analysis:  Are  the  MSG’s  outputs 
easy  to  analyze  or  are  they  intended  for  use  in  subsequent  analyses? 

Comment: 


#141.  Assessment  -  Cost  Effectiveness:  Do  you  think  that  the  MSG 
represents  a  cost-effective  way  to  get  at  the  issues  it  addresses,  or 
would  you  recommend  alternative  procedures,  methods,  or  techniques? 

Comment : 
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#142,  Assessment  -  Validation  Criteria:  What  questions  related  to 
validation  have  been  posed,  and  are  they  clear  and  concise  or  are  they 
vague,  confusing  or  non-existent?  Has  much  attention  been  given  to 
validation  of  the  MSG? 

Comment : 


#143.  Assessment  -  Validation:  Based  on  the  criteria  that  were 
developed,  was  any  validation  done  on  the  MSG?  What  resulted? 

Comment: 


#144.  Assessment  -  Overall:  In  your  opinion  is  the  MSG  of 
outstanding,  average,  poor,  or  of  indeterminate  quality?  Would  you 
commend  it  for  future  use?  Unqualified  acceptance?  Qualified? 


Comment: 


Appendix  B 

MODELS,  SIMULATIONS.  AND  GAMES  SURVEYED 


FIRST  VERSION  OF  QUESTIONNAIRE 
XRAY 

Inter-Nation  Simulation 

Business  Game  for  Teaching  and  Research  Purposes 

SECOND  VERSION  OF  QUESTIONNAIRE 

QUICK  (Quick  Reacting  General  War  Gaming  System) 

GO 

FAST-VAL 

Division  Battle  Model 
CAIR-COMP 

SIDAC  (Single  Integrated  Damage  Analysis  Capability) 

Supply  Point  Simulation  Model 

SIGMALOG  Theater  Materiel  Model 

TCM  (Theater  Combat  Force  Requirements  Model) 

LP-2  (Laboratory  Problem  2) — ICBM  Logistics  Simulation 

CARMONETTE 

CASCADE  III 

SATAW 

FINAL  VERSION  OF  QUESTIONNAIRE 

Time  Reported 
To  Complete 
Questionnaire  (hr) 

1.  LEGAL  MIX  IV  .  1 

2.  EFFECT  3  &  4  .  5 

3.  SMILEY  .  13 

4.  NOMOS -NOMOGRAPHS  .  12 

5.  VALUAT  V  .  14 

6.  Cost  Effectiveness  Model  .  6 

7.  Evacuation  Policy  Model  .  3.5 
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Time 

8.  TAM  (Target  Acquisition  Model)  .  8 

9.  DYNTACS  X — Small-Unit  Combat  Simulation  .  2.5 

10.  HOVARM  .  7 

11.  New  Unit  Cost  Model,  Modified  . 4 

12.  DYNTACS  (Dynamic  Tactical  Simulation)  .  3.5 

13.  STANO — System  Assessment  Model,  Phase  1  (SAM  I)  . 8 

14.  Trans-Hydro  Craft  .  7 

15.  Fire  Support  Simulation  . 5 

16.  APSUB  MOD  0-1  .  1-5 

17.  CFOAM/TACOPS  (Tactical  Operations  Model  of  Continuous 

Fleet  Operations)  .  7 

18.  APAIR  .  8 

19.  APSURF  .  2-5 

20.  LOTRAK.  II — ASW  Localization  Model  .  5 

21.  AS  GRAM .  5 

22.  Nuclear  Exchange  Model  III  .  3 

23.  STEM  (Sub  Trailing  Evaluation  Model)  .  4 

24.  POSTURE  . . 

25.  SUBDUEL  1  .  4 

26.  VERS  (Vehicle  and  Equipment  Requirements)  .  8 

27.  SASWEM  (Sub  ASW  Engagement  Model)  .  15 

28.  Detailed  Ship  Loading  .  8 

29.  VEMPBOLT  (Vehicle  Mission  Processor  Based  on  Least  Times)  5 

30.  TATAWS  .  3 

31.  FLAME  I  .  13- 5 

32.  EX0-1  .  5 

33.  FAIR  PASS  .  H-5 

34.  LOADER  .  3 

35.  SPARE  (Strike  Planning,  Aircraft  Requirements  Evaluator)  .  5 

36.  SIMCAT  (Simulation  of  Contingency  Air  Terminal  Model)  ....  3 

37.  ATLAS  . 6 

38.  FAME  .  5 

39.  ASWAS  (ASW  Air  System  Model)  .  5 

40.  ATMIX — Individual  Unit  Action  .  4 
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Time 

41.  CODE  50  (Navy)  .  3 

42.  SAAB  (Simulated  Air-to-Air  Battle)  .  3 

43.  MARADS  .  9 

44.  ASESEM  (ASW  Escort  Engagement  Model)  .  6.5 

45.  SPOL  (Shore  Party  Operation  and  Logistics)  . 4 

46.  SWEM  (Strategic  Weapon  Exchange  Model)  . 10 

47.  CAM-SAM  .  3.5 

48.  Minefield  Analysis  with  Hunting  Evaluation  Model  .  5 

49.  STS-2  (Strip-to-Shore  Model)  .  6 

50.  SAMEN  .  5 

51.  ORION  . 14 

52.  LEGION  .  20 

53.  TARTARUS  IV  N/C0C0  .  16 

54.  COBRA  (Comprehensive  Blast  and  Radiation  Assessment 

System)  . NR 

55.  AEM  (Arsenal  Exchange  Model)  .  4 

56.  SINUS  (Small  Infantry  Unit  Simulator,  formerly 

CARMONETTE)  .  2 

57.  ETNAM  .  1 

58.  SOAR  .  2 

59.  TRACS  .  1 

60.  TRAMPS  .  1 

61.  M0RG  . .  2 

62.  MACE  .  1 

63.  THEFT  .  1.5 

64.  Localization  LASCAR  . 9 

65.  NETWORK  SIMULATOR  .  9.5 

66.  ASWASP  .  8 

67.  AAWSEM  (Anti-Air  Warfare  System  Effectiveness  Model)  .  6 

68.  TACOS  II  .  32 

69.  DMEW  (Deterministic  Mix  Evaluation,  Worldwide)  . .  24 

70.  GRAPHICS  .  32 

71.  GFE  III  (Gross  Feasibility  Estimator)  . .  14 

72.  VALIMAR  .  NR 
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Time 

73.  AREA  DOMINATION  II  .  4 

74.  TAR  III  (Target  Acquisition  Routine)  .  15 

75.  FORECAST  II  .  27 

76.  TAFCOM  .  4 

77.  SAM  .  4.5 

78.  Air  Contingency  Terminal  Simulator  . 4 

79.  GIANT  .  8 

80.  TACAIR  .  69 

81.  OASIS  -  71  . 28 

82.  PEGASOS  (Penetration  Evaluation  Gaming  Analysis 

Strategic  Offensive  Studies)  .  24 

83.  Advanced  Penetration  Model  .  3 

84.  WEAPON  .  24 

85.  TAC  AVENGER  .  10 

86.  COMBO — Combat  Bombing  Weaponeering  Program . . .  4.5 

87.  MPASS  .  13 

88.  SADDLE  .  19.5 

89.  ATOM  (Air  Tactical  Operations  Model)  .  4 

90.  ENDO-1  .  5.5 

91.  "Candidate  Families"  Methodology:  Simulation,  Cost, 

and  Optimization  Models  .  48 

92.  ATLAS  (Tactical,  Logistical,  and  Air  Simulation)  .  40 

93.  Global  Distance  Routine  . .  10 

94.  INFANTRY  .  69 

95.  SPHINX  II  (Survival  Probability  Hazard  in  a  Nuclear 

Exchange)  .  25 

96.  SOUCA — Soviet  Capabilities  .  14 

97.  NEWCON  .  80 

98.  FASTALS  (Force  Analysis  Simulation  of  Theater 

Administrative  and  Logistics  Support)  . . 10 

99.  SOURCE  .  8 

100.  SDPS  (Space  Defense  Planning  Simulator)  . 19.5 

101.  OFD  (Objective  Force  Design)  . 20 

102.  PFD  (3-Preliminary  Force  Design)  . . 13 
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Time 

103.  DELOGREQ  (Deployment  Logistics  Requirements  Model)  ......  4 

104.  PFD-SAM  (Preliminary  Force  Designer  Intertheater 

Movements  Simulation)  . 35 

105.  BOMB  SIM  .  4 

106.  UNCLE  .  3 

107.  STAR-III  . 6.5 

108.  OPSTRAS  .  NR 

109.  CEM  (Campaign  Execution  Model)  . . .  4 

110.  SWIM  II  .  3.5 

111.  DIVWAG  .  3 

112.  APSURV  Model  1  .  2 

113.  ADM  (Atomic  Demolition  Munition)  .  14 

114.  SEANITE0PS  AGGREGATE  COST  MODEL  .  5 

115.  THEATERSP I EL  .  8 

116.  TBM  (Theater  Battle  Model)  . . .  4 

117.  TACSPIEL  (Computerized)  . 16 

118.  GLOBAL  .  8 

119.  ADVICE  II  .  5 

120.  DIVTAG  II  .  3 

121.  DBM  (Division  Battle  Model)  . . . .  6 

122.  COMMEL  .  40 

123.  HOVER  .  9 

124.  TEEM  (TACFIRE  Effectiveness  Evaluation  Model)  . .  20 

125.  Modif ied-Filter  . 10 

126.  SUB-AIR  BARRIER  .  4 

127.  Strategic  Force  Mix  Model  . . .  5 

128.  Corps  Battle  Model  .  8 

129.  SYNTAC  .  28 

130.  APCAMP  (ASW  Program  Campaign  Model)  . 3.5 

131.  SIRNEM .  8 

132.  Multi-Ship  ASW  Simulation  .  4.5 

133.  VIM  (Not  included  in  evaluation)  . .  . 


Appendix  C 

SURVEY  CQDEBQOK  AND  LIST  OF  VARIABLES 


Card  1: 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

Office 

Card 

Integer 

1-2 

Office  Use 

ID 

Sequence  number  of  the 

Integer 

3-5 

observation 

Identification 

2 

tiSGNME 

MSG  Name 

Text 

6-30 

7A 

RESUME 

Respondent's  Name 

Text 

31-50 

7B 

RESTIT 

Respondent's  Title 

Text 

51-80 

Card  2: 


Office 

CARD 

Integer 

1-2 

Office  Use 

ID 

Sequence  number  of 
observation 

Integer 

3-5 

Identification 

28 

DESIGN 

Designing  or  building 
Agency  or  firm 

Text 

6-30 

2C 

AUTHOR 

Author's  name  of  MSG 

Text 

31-50 

2D 

SPONSR 

Sponsor  responsible  for 
Initiation 

Text 

51-70 

2E 

AGENCY 

Agency  ordering  the  work 

Text 

71-80 
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Card  3: 

— - - 

General 

Class 

Variable 

Number 

Vari able 

Name 

Definition 

Attributes 

Column 

Of f i ce 

CARD 

Integer 

1-2 

Office  Use 

ID 

Sequence  number 

Integer 

3-5 

1 

QT 

Questionnaire  Time:  hours 
to  complete  questionnaire 

Real  number  l 

6-9 

Descri  ption 

3 

CATEG1 

Designation  of  item  as  MSG 

0  -  Model  i 

1  -  Simulation 

2  -  Man -Machine 

3  -  Manual 

4  -  Analysis  or 

o-n*[2; 

Study 

5  -  Other 

9  -  No  response 

4 

RESR0L1 

Respondent  (to  questionnaire) 

0  -  Funder/sponsor 

12-14  [3 

role 

1  -  User 

2  -  Design/build 

3  -  Player 

4  -  Caretaker 

5  -  Control 

6  -  Other 

9  -  No  response 

5 

RTIME 

Length  of  time  respondent 

Real  Number 

15-16 

acquainted  with  work,  ex¬ 
pressed  in  months 

10 

PURP1 

Major  stated  purpose  of  the 

0  -  Tech.  Eval . 

17-19  [ 

work 

1  -  Doctrinal 

Eval . 

2  -  Force  Structure 

3  -  PDM/IR 

4  -  PME/Domestic 

5  -  T/E 

6  -  Research/Metho 

7  -  Other 

9  -  No  Response 

d 

10CL 

Confidence  level  for  PURP 

0  -  No  Response 

1  -  Low 

2  -  Middle 

3  -  High 

4  -  Absolutely 

20 

certain 

*First  or  dominant  choice  goes  in  first  column;  second  goes  into  second,  etc 
field  are  left  blank.  [2]  «  Two  possibilities. 


If  only  one  choice,  remaining  columi 


i 
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Card  3 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Col umn 

Description, 

cont. 

11 

CLASWO 

Classification  of  MSG 
(without  inputs) 

0  -  NA;Uncl ass. 

1  -  FOUO 

2  -  Confid. 

3  -  Secret 

4  -  TS 

5  -  Proprietary 

6  -  Other 

9  -  No  Response 

21 

12 

CLASWI1 

Classification  of  MSG 
when  data  added 
(highest  noted  first) 

0  -  NA;Unclass. 

1  -  FOUO 

2  -  Confid. 

3  -  Secret 

4  -  TS 

5  -  Proprietary 

6  -  Other 

9  -  No  Response 

22-23  [2] 

13A 

PROREV 

Has  MSG  been  given  profes¬ 
sional  external  review? 

0  -  No 

1  -  Yes 

9  -  No  Response 

24 

13B 

REVDAT 

How  many  months  since  last 
professional  review  of 

MSG? 

0  -  Not  done 

1  -  <6  months 

2  -  6-12  months 

3  -  13-24  months 

4  -  25-36  months 

5  -  >36  months 

9  -  No  Response 

25 

14 

MSGPAR 

Number  of  MSG  parents 
(direct) 

0  -  None 

1  -  One 

2  -  More  than  one 

9  -No  Response 

26 

14CL 

Confidence  level  for  MSGPAR 

(See  Col.  20) 

27 

19 

MSGPRO 

Number  of  MSG  spinoffs 

0  -  None 

1  -  One 

2  -  More  than  one 

9  -  No  response 

28 

17 

(17  &  18) 

DEVTM1 

Total  time  elapsed  between 
development  and  initiation 
expressed  in  months 

0  -  <3  months 

1  -  3-6  months 

2  -  7-12  months 

3  -  13-18  months 

4  -  19-24  months 

5  -  26-30  months 

6  -  31-36  months 

7  -  37-42  months 

8  -  *43  months 

9  -  No  Response 

29 
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Card  3 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

MSG  Production  and 
Purpose 

21 

WH01 

Source  of  funds  to  build  MSG 

0  -  University 

1  -  Foundation 

2  -  Private 

3  -  ARPA 

4  -  JCS 

5  -  USA 

6  -  USAF 

7  -  USN 

8  -  Other  DOD 

9  -  Other  U.S.Gov. 

30-33  [4] 

22 

WH0DID1 

Who  produced  MSG? 

0  -  Private 

1  -  University 

2  -  For  profit 

3  -  Armed  Forces 

4  -  Hot  for  Profit 

9  -  No  Response 

34-35  [2] 

23 

INITIA 

Who  initiated  the  model? 

Who  was  responsible  for 
getting  work  underway? 

0  -  NA;Unknown 

1  -  Builders/research 

2  -  In  house/User 

3  -  In  house/non-user 

4  -  External/user 

5  -  External/non-user 

9  -  No  Response 

36 

24 

t 

INPURP1 

1 

What  was  the  initiator’s 
(Variable  #23)  purpose 

0  -  NA; Unknown 

1  -  Teach/Trng. 

2  -  Anal/Diag. 

3  -  Ops. 

4  -  Exper. 

5  -  Research/theory 

6  -  Advocacy 

7  -  Other 

9  -  No  Response 

37-39  [3] 

24CL 

Confidence  level  for  INPURP 

40 

25 

SPCPRP 

How  specific  was  funding 
source  in  designation  of  its 
purpose? 

0  -  NA; Unknown 

1  -  Tight 

2  -  Moderate 

3  -  Diffuse 

9  -  No  Response 

41 

25  CL 

Confidence  level  for  SPCPRP 

42 

26 

ALTPR01 

Best  alternative  procedures 
to  the  MSG 

0  -  NA.  :Unknown 

1  -  None 

2  -  Lectures 

3  -  Cases/history 

4  -  Analysis 

5  -  Experience 

6  -  Gaming 

9  -  No  Response 

43-44  [2] 
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Card  3 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Col umn 

MSG  Production 
and  Purpose, 
cont. 

27 

USE1 

What  has  been  the  major 
use  of  MSG?  [Order  with 
respect  to  #24] 

Same  as  #24 

45-47  [3] 

27CL 

Confidence  level  for  USE 

48 

30 

BRIEF 

Number  of  briefings  based 
on  the  MSG 

0  -  0 

1  -  1-3 

2  -  4-6 

3  -  7-9 

4  -  10-12 

5  -  13-15 

6  -  16-18 

7  -  19-21  or  more 

8  -  Unknown 

9  -  No  Response 

49 

MSG  Costs 

35 

CSTDIR 

Direct  costs  to  build 

O 

A 

■f* 

7* 

50 

to  ouild 

1  -  50-99K 

2  -  100-249K 

3  -  250-499K 

4  -  5QQ-749K 

5  -  750-999K 

6  -  1-2.49  Million 

7  -  2.5-5  Million 

8  r  >5  Million 

9  -  NA;  Unknown; 

No  Response 

35CL 

Confidence  level  for  CSTDIR 

■„ 

51 

35A 

CSTFAt: 

Direct  costs  for  all  in 

Same  as  #35 

52 

35ACL 

series 

Confidence  level  for  CSTFAM 

53 

36 

DIRFOS 

Direct  funds  used  to  build? 

0  -  No 

54 

1  -  Yes 

37 

FUNDI  R 

Direct  funding  to  build 

9  -  No  Response 

Same  as  #35 

55 

37CL 

Confidence  level  for  FUNDIR 

56 

38 

38CL 

TOTCST 

Total  of  all  types  of  cost 
to  build  MSG 

Confidence  level  for  TOTCST 

Same  as  #35 

57 

58 

MSG  Costs  to 

39 

CSTOPS 

Average  annual  operating 

0  -  <9K 

59 

Operate 

cost 

1  -  1 0-24  K 

2  -  25-49K 

3  -  50-74K 

4  -  75-99K 

5  -  >100K 

9  -  No  Response 
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Card  3 — continued 


General 

Variable 

Variable 

Attributes 

Column 

Class 

Number 

Name 

Definition 

MSG  Costs  to 

39  CL* 

Confidence  level  for  CSTOPS 

60 

Operate,  cont. 

40 

CSTRUN 

Operating  cost  for  a  single 

Same  as  #39 

61 

run 

41 

CSTUP 

Annual  update  costs 

Same  as  #39 

62 

41  CL 

Confidence- level  CSTUP 

63 

42 

0PLIF1 

Operational  life  span  of 

MSG,  in  months 

Same  as  #17 

64 

42CL 

Confidence  level  for  0PLIF1 

65 

43 

ACTIVE 

Still  operational? 

0  -  No 

66 

1  -  Yes 

9  -  No  Response 

43CL 

Confidence  level  for  ACTIVE 

67 

45 

0PFRQ1 

Operational  use  frequency 

0  -  Not  operational 

68 

per  year  (average) 

1-1-5 

2  -  6-10 

3  -  11-15 

4  -  16-20 

5  -  21-25 

6  -  26-50 

7  -  51-100 

8-101  or  more 

9  -  No  Response 

46 

EXP USE 

Used  for  experimental 

0  -  No 

69 

purposes? 

1  -  Yes 

9  -  No  Response 

48 

EXPRP 

Was  the  MSG  intended  to 
be  used  experimentally? 

Same  as  above 

49 

EDUSE 

Used  for  educational 
purposes? 

Same  as  above 

51 

EDPRO 

Was  the  MSG  intended  to 
be  used  educationally? 

Same  as  above 

52 

TRANSU 

Level  of  difficulty  of 

0  -  Not  for  transfer 

transferal  of  MSG 

1  -  Generally 

2  -  Moderate 

3  -  Middling 

4  -  Difficult 

5  -  Extreme 

9  -  No  response 

Card  3 — continued 

General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

MSG  Costs  to 
Operate,  cont. 

53 

TRAN SC 

Transfer  costs  to  operate 

0  -  NA; Unknown 

1  -  Cost 

2  -  +10% 

3  -  +10-25% 

4  -  +25-50% 

5  -  +50-100% 

6  -  >100% 

74 

53CL 

Confidence  level  for  TRAHSC 

75 

55 

DUPMSG 

Is  there  an  MSG  that  serves 
same  purpose? 

0  -  No 

1  -  Yes 

9  -  No  Response 

76 

Recommendations 

Opinions 

57 

CLEAR 

| 

Clearinghouse  utility 

0  -  Highly  useful 

1  -  Useful 

2  -  Same 

3  -  Harmful 

4  -  Highly  harmful 

9  -  No  Response 

77 

58 

STNRD1 

Standardization's  utility 

0  -  Highly  useful 

1  -  Useful 

2  -  Same 

3  -  Harmful 

4  -  Highly  Harmful 

9  -  No  Response 

78 

59 

REGCEN 

Operation  utility  of 
regional  center 

[coded  from  verbal  response] 

Same  as  above 

79 

60 

EXREV 

Opinion  on  utility  of 
external  review  board 

[coded  from  verbal  response] 

Same  as  above 

80 

Card  4: 


General 

Variable 

Variable 

Definition 

Attributes  ! 

Column 

Class 

Number 

Name 

Off i ce 

CARD 

Integer 

1-2 

Office  Use 

ID 

Sequence  number 

Integer 

3-5 

riSG  Characterization 

RES2 

Is  the  technical  section 

0  -  Ho 

6 

and  Description 

filled  out  by  same  person 

1  -  Yes 

(Techni cal ) 

as  the  initial  segment 

62 

SCENTP 

Scenario  type  of  MSG 

0  -  Numerical 

1  -  Verbal  for  use 

2  -  Verbal  for  Analysis 

9  -  No  Response 

7 

64 

MATHS 

Mathematical  level  of 

0  -  None 

8 

sophistication  of  MSG 

1  -  Slight 

2  -  Moderate 

3  -  High 

9  -  No  Response 

65 

MOVES 

"Clock"  used  to  sequence 

0  -  NA: Unknown 

9 

MSG  moves 

1  -  Event 

2  -  Fixed 

3  -  Combination  1&2 

9  -  No  Response 

66 

MT/RT 

Model  time  to  real  time 

0  -  NA; Unknown 

10 

ratio 

1  -  Highly  compressed 

2  -  Compressed 

3  -  Real  Time 

4  -  Expanded 

5  -  Highly  expanded 

9  -  Ho  Response 

66a 

Confidence  Level  for 

MT/RT 

11 

67 

TIME 

Time  represented  in  MSG 

0  -  Past 

1  -  Present 

2  -  Future 

3  -  Unspecified 

4  -  Not  relevant 

5  -  Conti  nation  of  abov< 

9  -  No  Response 

12 

E 
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Card  4 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

MSG  Characterization 
and  Description, 
cont. 

68 

LRTIME 

Level  of  resolution,  model 
time,  smal lest  unit 

0  -  NA  ;Unknown 

1  -  Seconds 

2  -  Minutes 

3  -  Hours 

4  -  Days 

5  -  Weeks 

6  -  Quarters 

7  -  Years 

8  -  >Years 

9  -  No  Response 

13 

69 

LRSPCE 

Level  of  resolution,  smallest 
model  spatial  count  of 
analysis 

0  -  NA: Unknown 

1  -  Meters 

2  -  Kilometers 

3  -  Theater/ continent 

4  -  Varied 

9  -  No  response 

14 

69CL 

Confidence  Level  for  LRSPCE 

15 

70 

LRSIDE 

Level  of  resolution,  sides  in 
MSG 

0  -  NA; Unknown 

1  -  Individuals 

2  -  Small  groups (struct.) 

3  -  Small  orgs. 

4  -  Large  orgs. 

5  -  Very  large  orgs. 

6  -  Small  groups (unstruct. ] 
9  -  No  Response 

16 

71 

LRMIL 

Level  of  resolution,  military 
action 

0  -  NA;Unknovm 

1  -  Engagement 

2  -  Battle 

3  -  Campaign 

4  -  War 

5  -  Diplomatic 

17 

71CL 

Confidence  level  for  LRMIL 

18 

72 

RNDM 

Are  random  events  considered? 

0  -  No 

1  -  Yes 

9  -  No  Response 

19 

MSG  Planning  Factors 
ana  data 

73 

DATAS 

Where  did  data  come  from? 

0  -  NA; Unknown 

1  -  Military/not  cross¬ 

checked 

2  -  Military/cross-checked 

3  -  Civil 

4  -  Generated  own 

5  -  Field  exercise 

6  -  Combination  of  above 

9  -  Ho  Response 

20 
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Card  4 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

MSG  Planning 
Factors  and 

Data,  cont. 

78 

SENSET 

Was  sensitivity  analysis  done? 

0  -  No 

1  -  Yes 

9  -  No  Response 

21 

80 

DATAV 

Who  validated  the  data,  how 
well  [coded  from  text] 

0  -  NA;Unknown 

1  -  High  quality 

2  -  Moderate 

3  -  Weak 

4  -  Not  done 

9  -  No  Response 

22 

82 

LANG 

What  language  is  MSG 
written  in? 

0  -  NA 

1  -  FORTRAN 

2  -  PL-1 

3  -  COBOL 

4  -  GPSS 

5  -  SIMSCMPT 

6  -  ALGOL 

7  -  ASSEMBLER 

8  -  Other 

9  -  No  Response 

23 

82CL 

Confidence  level  for  LANG 

24 

83 

MSGS I 2 

How  many  computer  instruc¬ 
tions  (1000' s)? 

0  -  NA: Unknown 

1  -  <1000 

2  -  1 -2K 

3  -  2-4K 

4  -  4-6K 

5  -  6-8K 

6  -  8-1  OK 

7  -  >10K 

25 

* 

9  -  No  Response 

84 

FAC1L 

Special  facilities  needed 
for  MSG 

0  -  NA; Unknown 

1  -  Special  building 

26 

2  -  Dedicated  computer 
(unclas.) 

3  -  Dedicated  computer 
(classi fied) 

4  -  Special  language, 
library,  or  system 

9  -  No  Response 

84CL 

Confidence  level  for  FACIL 

27 

MSG  Production 
cos  ts 

85 

85CL 

DEVTM2 

Development  time,  in  months 

Confidence  level  for  DEVTM2 

Same  as  #17 

28 

29 

80 

DEVMY7 

1 

Total  man  years  in  develop¬ 
ment 

0  -  NA; Unknown 

1  -  0-1 

2  -  2-5 

3  -  6-10 

4  -  11-20 

5  -  >20 

9  -  No  Response 

30 
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Card  4 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

MSG  Production 

Costs,  cont. 

86CL 

Confidence  level  for  DEVMY1 

31 

87 

DVPMY1 

Total  professional  man-years 
in  development 

Same  as  #86 

32 

88CL 

Confidence  level  for  DVPMY1 

33 

89 

DVPRG1 

Total  programmer  man-years 
in  development 

Same  as  #86 

34 

MSG 

Documentation 

90 

DOCEXT 

Extent  of  documentation, 
general  assessment 

0  -  NA; Zero: Unknown 

1  -  Excellent 

2  -  VG 

3  -  Average 

4  -  Weak 

5  -  Poor 

6  -  Uneven/variable 

7  -  Unavailable 

8  -  Combinations  of  above 

9  -  No  Response 

35 

91 

D0CL0C 

Where  is  the  documentation 
located? 

0  -  NA; Unknown 

1  -  Out  of  print/Unk. 

2  -  Proprietary  /NFP 

3  -  Proprietary/Commercial 

4  -  Proprietary/Author 

5  -  Prop. (Class. )/Author 

6  -  Public/Doc 

7  -  Public/Loc 

8  -  Public/Ooumals,  books 

9  -  No  Response 

36-38  [3] 

93 

DOCPUB 

Publication  type  of 
documentati  on 

0  -  Books/articles 

1  -  User  manual 

2  -  Program  deck 

3  -  Reports 

4  -  0  &  1 

5  -  1  &  2 

6  -  Other  combinations 

9  -  No  Response 

39 

Techni  cal 

Reconmendations/ 

Opinions 

95 

TECH 

Technical  coordination 
opinion 

0  -  Highly  undesirable 

1  -  Undesirable 

2  -  Indifferent 

3  -  Desirable 

4  -  Highly  desirable 

9  -  No  Response 

40 
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Card  4 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

Technical  Rec¬ 
ommendations,  cont. 

96 

STNRD2 

Standards  Committee 
[coded  from  verbal  text] 

Same  as  *58 

41 

Questionnaire 

Evaluation 

97 

QUEVAL 

Respondent  evaluation 
of  questionnaire 

0  -  Excellent 

1  -  Good 

2  -  Modal 

3  -  Poor 

4  -  Bad 

9  -  No  response 

42 

98 

IQC 

Researchers '  assessment  of 
the  quality  of  the  question¬ 
naire  responses 

Same  as  above 

43 

136 

VOLAS 

Did  the  respondent  provide 
information  in  the  voluntary 
assessment  portion  of  the 
questionnaire? 

0  -  No 

1  -  Yes 

44 

Types  of  data 

74 

DATAT 

Types  of  data 

0  -  NA;Unknown 

1  -  Type  1  only 

2  -  Type  2  only 

3  -  Type  3  only 

4  -  Types  1  and  2 

5  -  Types  1  and  3 

6  -  Types  2  and  3 

7  -  All  Types 

9  -  No  response 

45 

Intangibles: 
Assumptions  made 
or  utilized 

77 

INTASP 

Intangibles: 

Assumptions 

Hade  in  Model 

0  -  No 

1  -  Yes 

2  -  Could  not 

determine 

9  -  No  response 

46 

Service 

SERVIC 

Service  Using  the  MSG 

1  -  USA 

2  -  USAF 

3  -  USN 

4  -  Other  DOD 

47 

Use  Date 

18 

USEDAT 

USE  initiation  date 

0  -  Never  operated 

1  -  <57 

2  -  58-59 

3  -  60-61 

4  -  62-63 

5  -  64-65 

6  -  66-67 

7  -  68-70 

8  -  71-72 

9  -  NA,  Unknown, 

No  response 

48 

158- 


Card  5: 


General 

Variable 

Variable 

Class 

Number 

Name 

Definition 

Attributes 

Column 

Office 

Card 

Integer 

1-2 

Office  Use 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

Production 

35* 

CSTDR* 

Direct  costs  to  build 

Real 

6-10 

Costs 

(1000's) 

35A$ 

CSTFH$ 

Direct  costs  to  build 
family  of  models 
(1000 1 s ) 

Real 

11-15 

37* 

FUNDRS 

Direct  funding  to  build 
(1000's) 

Real 

16-20 

38$ 

TOTS 

Total  of  all  types  of 
costs  to  build  MSG 
(1000' s) 

Real 

21-25 

Operational 

39$ 

CSTOPS 

Average  annual  operating 

Real 

26-30 

Costs 

costs  (1000's) 

40$ 

CSTRN$ 

Cost  per  single  run 
(1000's) 

Real 

31-35 

41$ 

CSTUP$ 

Annual  update  costs 
( 1000 1 s ) 

Real 

36-40 

42  R 

0PLIF2 

Number  of  months 

MSG  operational 

Integer 

41-42 

45R 

0PFRQ2 

Number  of  times  (total) 

MSG  has  been  run  for  any 

5  years 

Integer 

43-45 

Data 

75  A 

IN  CON 

Number  of  input  constants 

Integer 

46-48 

kequi rements/ 

Si  ze 

758 

INPAR 

Number  of  input  parameters 

Integer 

49-51 

75C 

INVAR 

Number  of  input  variables 

Integer 

52-54 

76 

OUTVAR 

Number  of  output  variables 

Integer 

55-57 

Development 

85R 

DEVTM3 

Number  of  months  MSG  under 

58-59 

Time 

development 

Integer 

86k 

UEVMY2 

Number  of  total  man-years 
under  development 

Integer 

60-63 

87R 

1 

DVPMY2 

Number  of  professional 
man-years  under  develop¬ 
ment 

Integer 

64-67 

89  R 

DVPRG2 

Number  of  programmer 
man-years  under  develop¬ 

Integer 

68-71 

% 

ment 

Cards  6-12  (narrative  descriptions) : 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

Card  6: 

Office 

Card 

Integer 

1-2 

Office  Use 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

Purpose 

9 

PURPV 

Verbal  summary  of 
specific  purposes 

Hollerith 

6-80 

Card  7: 

Offi ce 

Card 

Integer 

1-2 

Off i ce 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

An tecedents/ 
Spinoffs 

14 

ANTEC 

Names  of  antecedents 

Hoi leri th 

6-40 

19 

SPIN 

Names  of  spinoffs 

Holleri th 

41-80 

Card  8: 

Office 

Card 

Integer 

1-2 

Office 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

Benefits/ 

Use 

31 

BRIEF 

Who  was  briefed? 

Hollerith 

6-40 

32 

PRPBRF 

What  was  the  specific 
purpose  of  the  briefing? 

Hollerith 

41-80 

Card  9: 

Office 

Card 

Integer 

1-2 

Office 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

33 

IMPBRF 

Importance  of  the  MSG  to 
<ieci  sions 

Hollerith 

6-40 

34 

BNFIT 

Criteria  listed  as 
benefits  for  having  used 
this  particular  MSG 

Hollerith  . 

Cards  6-12 — continued 


General 

Class 

Variable 

Number 

Variable 

Name 

Definition 

Attributes 

Column 

Card  10: 

Office 

Card 

Integer 

1-2 

Office 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

44 

USERS 

Who  uses  the  MSG  at  this 
time? 

Hollerith 

6-40 

54 

OBSOL 

Estimated  speed  at  which 
respondent  thinks  the  model 
will  become  obsolete;  reasons 
for  this,  if  noted 

Hoi  leri  th 

41-80 

Card  11: 

Off i ce 

Card 

Integer 

1-2 

Office 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

Data 

73A 

DAT  VI 

Sources  of  the  data  used 
in  the  MSG 

Hollerith 

6-40 

I 

80A 

DATV2 

Procedures  used  to  validate 
the  data 

Holleri th 

41-80 

Card  12: 

Of f i ce 

Card 

Integer 

1-2 

Office 

ID 

Sequence  number  of  the 
observation 

Integer 

3-5 

Languages/ 

Machinery 

81 

LANGS 

Languages  used  to  code 
this  MSG 

Hollerith 

6-40 

82  A 

COMPS 

Computers  on  which  this 

MSG  is  (has  been)  run 

Hollerith 

41-80 

00 

cr 

c 

cr 

H* 

7T 

05 

5 

CL 

n 

Q 

£ 

fD 
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O 

0s 

o 

I 

> 

pc 

*d 

> 

pc 

o 


MODELS,  SIMULATIONS,  AND  GAMES-A  SURVEY 


